TransUNet: Rethinking the U-Net architecture design for medical image segmentation through the lens of transformers.

Medical image segmentation is crucial for healthcare, yet convolution-based methods like U-Net face limitations in modeling long-range dependencies. To address this, Transformers designed for sequence-to-sequence predictions have been integrated into medical image segmentation. However, a comprehensive understanding of Transformers' self-attention in U-Net components is lacking. TransUNet, first introduced in 2021, is widely recognized as one of the first models to integrate Transformer into medical image analysis. In this study, we present the versatile framework of TransUNet that encapsulates Transformers' self-attention into two key modules: (1) a Transformer encoder tokenizing image patches from a convolution neural network (CNN) feature map, facilitating global context extraction, and (2) a Transformer decoder refining candidate regions through cross-attention between proposals and U-Net features. These modules can be flexibly inserted into the U-Net backbone, resulting in three configurations: Encoder-only, Decoder-only, and Encoder+Decoder. TransUNet provides a library encompassing both 2D and 3D implementations, enabling users to easily tailor the chosen architecture. Our findings highlight the encoder's efficacy in modeling interactions among multiple abdominal organs and the decoder's strength in handling small targets like tumors. It excels in diverse medical applications, such as multi-organ segmentation, pancreatic tumor segmentation, and hepatic vessel segmentation. Notably, our TransUNet achieves a significant average Dice improvement of 1.06% and 4.30% for multi-organ segmentation and pancreatic tumor segmentation, respectively, when compared to the highly competitive nn-UNet, and surpasses the top-1 solution in the BrasTS2021 challenge. 2D/3D Code and models are available at https://github.com/Beckschen/TransUNet and https://github.com/Beckschen/TransUNet-3D, respectively.

DMSPS: Dynamically mixed soft pseudo-label supervision for scribble-supervised medical image segmentation.

High performance of deep learning on medical image segmentation rely on large-scale pixel-level dense annotations, which poses a substantial burden on medical experts due to the laborious and time-consuming annotation process, particularly for 3D images. To reduce the labeling cost as well as maintain relatively satisfactory segmentation performance, weakly-supervised learning with sparse labels has attained increasing attentions. In this work, we present a scribble-based framework for medical image segmentation, called Dynamically Mixed Soft Pseudo-label Supervision (DMSPS). Concretely, we extend a backbone with an auxiliary decoder to form a dual-branch network to enhance the feature capture capability of the shared encoder. Considering that most pixels do not have labels and hard pseudo-labels tend to be over-confident to result in poor segmentation, we propose to use soft pseudo-labels generated by dynamically mixing the decoders' predictions as auxiliary supervision. To further enhance the model's performance, we adopt a two-stage approach where the sparse scribbles are expanded based on predictions with low uncertainties from the first-stage model, leading to more annotated pixels to train the second-stage model. Experiments on ACDC dataset for cardiac structure segmentation, WORD dataset for 3D abdominal organ segmentation and BraTS2020 dataset for 3D brain tumor segmentation showed that: (1) compared with the baseline, our method improved the average DSC from 50.46% to 89.51%, from 75.46% to 87.56% and from 52.61% to 76.53% on the three datasets, respectively; (2) DMSPS achieved better performance than five state-of-the-art scribble-supervised segmentation methods, and is generalizable to different segmentation backbones. The code is available online at: https://github.com/HiLab-git/DMSPS.

Entry of ZSWIM4 to the nucleus is crucial for its inhibition of KIT and BMAL1 in gastrointestinal stromal tumors.

BACKGROUND: Zinc finger SWIM-type containing 4 (ZSWIM4) is a zinc finger protein with its function largely uncharacterized. In this study, we aimed to investigate the role of ZSWIM4 in gastrointestinal stromal tumors (GISTs). RESULTS: We found that ZSWIM4 expression is inhibited by the predominantly mutated protein KIT in GISTs, while conversely, ZSWIM4 inhibits KIT expression and downstream signaling. Consistent with the observation, ZSWIM4 inhibited GIST cell survival and proliferation in vitro. RNA sequencing of GISTs from KITV558A/WT mice and KITV558A/WT/ZSWIM4-/- mice showed that loss of ZSWIM4 expression increases the expression of circadian clock pathway member BMAL1 which contributes to GIST cell survival and proliferation. In addition, we found that KIT signaling increases the distribution of ZSWIM4 in the nucleus of GIST cells, and which is important for its inhibition of KIT and BMAL1. In agreement with the results in vitro, the in vivo studies showed that ZSWIM4 deficiency increases the tumorigenesis of GISTs in KITV558A/WT mice. CONCLUSIONS: Taken together, our results revealed that the entry of ZSWIM4 to the nucleus is important for its inhibition of KIT and BMAL1, ultimately attenuating GIST tumorigenesis. The results provide a novel insight in the understanding of signal transduction in GISTs and lay strong theoretical basis for the advancement of GIST treatment.

Data Set and Benchmark (MedGPTEval) to Evaluate Responses From Large Language Models in Medicine: Evaluation Development and Validation.

Background: Large language models (LLMs) have achieved great progress in natural language processing tasks and demonstrated the potential for use in clinical applications. Despite their capabilities, LLMs in the medical domain are prone to generating hallucinations (not fully reliable responses). Hallucinations in LLMs' responses create substantial risks, potentially threatening patients' physical safety. Thus, to perceive and prevent this safety risk, it is essential to evaluate LLMs in the medical domain and build a systematic evaluation. Objective: We developed a comprehensive evaluation system, MedGPTEval, composed of criteria, medical data sets in Chinese, and publicly available benchmarks. Methods: First, a set of evaluation criteria was designed based on a comprehensive literature review. Second, existing candidate criteria were optimized by using a Delphi method with 5 experts in medicine and engineering. Third, 3 clinical experts designed medical data sets to interact with LLMs. Finally, benchmarking experiments were conducted on the data sets. The responses generated by chatbots based on LLMs were recorded for blind evaluations by 5 licensed medical experts. The evaluation criteria that were obtained covered medical professional capabilities, social comprehensive capabilities, contextual capabilities, and computational robustness, with 16 detailed indicators. The medical data sets include 27 medical dialogues and 7 case reports in Chinese. Three chatbots were evaluated: ChatGPT by OpenAI; ERNIE Bot by Baidu, Inc; and Doctor PuJiang (Dr PJ) by Shanghai Artificial Intelligence Laboratory. Results: Dr PJ outperformed ChatGPT and ERNIE Bot in the multiple-turn medical dialogues and case report scenarios. Dr PJ also outperformed ChatGPT in the semantic consistency rate and complete error rate category, indicating better robustness. However, Dr PJ had slightly lower scores in medical professional capabilities compared with ChatGPT in the multiple-turn dialogue scenario. Conclusions: MedGPTEval provides comprehensive criteria to evaluate chatbots by LLMs in the medical domain, open-source data sets, and benchmarks assessing 3 LLMs. Experimental results demonstrate that Dr PJ outperforms ChatGPT and ERNIE Bot in social and professional contexts. Therefore, such an assessment system can be easily adopted by researchers in this community to augment an open-source data set.

ScribSD+: Scribble-supervised medical image segmentation based on simultaneous multi-scale knowledge distillation and class-wise contrastive regularization.

Despite that deep learning has achieved state-of-the-art performance for automatic medical image segmentation, it often requires a large amount of pixel-level manual annotations for training. Obtaining these high-quality annotations is time-consuming and requires specialized knowledge, which hinders the widespread application that relies on such annotations to train a model with good segmentation performance. Using scribble annotations can substantially reduce the annotation cost, but often leads to poor segmentation performance due to insufficient supervision. In this work, we propose a novel framework named as ScribSD+ that is based on multi-scale knowledge distillation and class-wise contrastive regularization for learning from scribble annotations. For a student network supervised by scribbles and the teacher based on Exponential Moving Average (EMA), we first introduce multi-scale prediction-level Knowledge Distillation (KD) that leverages soft predictions of the teacher network to supervise the student at multiple scales, and then propose class-wise contrastive regularization which encourages feature similarity within the same class and dissimilarity across different classes, thereby effectively improving the segmentation performance of the student network. Experimental results on the ACDC dataset for heart structure segmentation and a fetal MRI dataset for placenta and fetal brain segmentation demonstrate that our method significantly improves the student's performance and outperforms five state-of-the-art scribble-supervised learning methods. Consequently, the method has a potential for reducing the annotation cost in developing deep learning models for clinical diagnosis.

Segmentation and Vascular Vectorization for Coronary Artery by Geometry-based Cascaded Neural Network.

Segmentation of the coronary artery is an important task for the quantitative analysis of coronary computed tomography angiography (CCTA) images and is being stimulated by the field of deep learning. However, the complex structures with tiny and narrow branches of the coronary artery bring it a great challenge. Coupled with the medical image limitations of low resolution and poor contrast, fragmentations of segmented vessels frequently occur in the prediction. Therefore, a geometry-based cascaded segmentation method is proposed for the coronary artery, which has the following innovations: 1) Integrating geometric deformation networks, we design a cascaded network for segmenting the coronary artery and vectorizing results. The generated meshes of the coronary artery are continuous and accurate for twisted and sophisticated coronary artery structures, without fragmentations. 2) Different from mesh annotations generated by the traditional marching cube method from voxel-based labels, a finer vectorized mesh of the coronary artery is reconstructed with the regularized morphology. The novel mesh annotation benefits the geometry-based segmentation network, avoiding bifurcation adhesion and point cloud dispersion in intricate branches. 3) A dataset named CCA-200 is collected, consisting of 200 CCTA images with coronary artery disease. The ground truths of 200 cases are coronary internal diameter annotations by professional radiologists. Extensive experiments verify our method on our collected dataset CCA-200 and public ASOCA dataset, with a Dice of 0.778 on CCA-200 and 0.895 on ASOCA, showing superior results. Especially, our geometry-based model generates an accurate, intact and smooth coronary artery, devoid of any fragmentations of segmented vessels.

Development of a CT-assessed adiposity nomogram for predicting outcome in localized ccRCC.

PURPOSE: This study aimed to develop and validate a computed tomography-based nomogram assessing visceral and subcutaneous adiposity for predicting outcomes in localized clear cell renal cell carcinoma (ccRCC). METHODS: A cohort of 364 patients with pathologically confirmed ccRCC participated in this retrospective study, with 254 patients assigned to the training set and 110 to the validation set (a 7:3 distribution ratio). The adipose score (AS) was generated using the least absolute shrinkage and selection operator Cox regression. Subsequently, a nomogram was constructed by integrating the clinical independent predictor with the AS to predict disease-free survival (DFS) in localized ccRCC after surgery. The performance of the nomogram was compared with the University of California, Los Angeles, Integrated Staging System (UISS), and the Stage, Size, Grade, and Necrosis (SSIGN) score. RESULTS: In both the training and validation cohorts, the nomogram exhibited superior discrimination compared to SSIGN and UISS (C-index: 0.897 vs. 0.781 vs. 0.776 in the training cohort, and 0.752 vs. 0.596 vs. 0.686 in the validation cohort; 5 year AUC: 0.907 vs. 0.805 vs. 0.820 in the training cohort, and 0.832 vs. 0.577 vs. 0.726 in the validation cohort). Decision curve analysis (DCA) revealed a superior net benefit across a wider range of threshold probabilities for predicting 5 year DFS compared to UISS and SSIGN scores. CONCLUSIONS: The developed prognostic nomogram demonstrated high accuracy and overall superior performance compared to existing prognostic models.

Evaluating and Enhancing Large Language Models' Performance in Domain-Specific Medicine: Development and Usability Study With DocOA.

BACKGROUND: The efficacy of large language models (LLMs) in domain-specific medicine, particularly for managing complex diseases such as osteoarthritis (OA), remains largely unexplored. OBJECTIVE: This study focused on evaluating and enhancing the clinical capabilities and explainability of LLMs in specific domains, using OA management as a case study. METHODS: A domain-specific benchmark framework was developed to evaluate LLMs across a spectrum from domain-specific knowledge to clinical applications in real-world clinical scenarios. DocOA, a specialized LLM designed for OA management integrating retrieval-augmented generation and instructional prompts, was developed. It can identify the clinical evidence upon which its answers are based through retrieval-augmented generation, thereby demonstrating the explainability of those answers. The study compared the performance of GPT-3.5, GPT-4, and a specialized assistant, DocOA, using objective and human evaluations. RESULTS: Results showed that general LLMs such as GPT-3.5 and GPT-4 were less effective in the specialized domain of OA management, particularly in providing personalized treatment recommendations. However, DocOA showed significant improvements. CONCLUSIONS: This study introduces a novel benchmark framework that assesses the domain-specific abilities of LLMs in multiple aspects, highlights the limitations of generalized LLMs in clinical contexts, and demonstrates the potential of tailored approaches for developing domain-specific medical LLMs.

TMO-Net: an explainable pretrained multi-omics model for multi-task learning in oncology.

Cancer is a complex disease composing systemic alterations in multiple scales. In this study, we develop the Tumor Multi-Omics pre-trained Network (TMO-Net) that integrates multi-omics pan-cancer datasets for model pre-training, facilitating cross-omics interactions and enabling joint representation learning and incomplete omics inference. This model enhances multi-omics sample representation and empowers various downstream oncology tasks with incomplete multi-omics datasets. By employing interpretable learning, we characterize the contributions of distinct omics features to clinical outcomes. The TMO-Net model serves as a versatile framework for cross-modal multi-omics learning in oncology, paving the way for tumor omics-specific foundation models.

Classification of lung cancer subtypes on CT images with synthetic pathological priors.

The accurate diagnosis on pathological subtypes for lung cancer is of significant importance for the follow-up treatments and prognosis managements. In this paper, we propose self-generating hybrid feature network (SGHF-Net) for accurately classifying lung cancer subtypes on computed tomography (CT) images. Inspired by studies stating that cross-scale associations exist in the image patterns between the same case's CT images and its pathological images, we innovatively developed a pathological feature synthetic module (PFSM), which quantitatively maps cross-modality associations through deep neural networks, to derive the "gold standard" information contained in the corresponding pathological images from CT images. Additionally, we designed a radiological feature extraction module (RFEM) to directly acquire CT image information and integrated it with the pathological priors under an effective feature fusion framework, enabling the entire classification model to generate more indicative and specific pathologically related features and eventually output more accurate predictions. The superiority of the proposed model lies in its ability to self-generate hybrid features that contain multi-modality image information based on a single-modality input. To evaluate the effectiveness, adaptability, and generalization ability of our model, we performed extensive experiments on a large-scale multi-center dataset (i.e., 829 cases from three hospitals) to compare our model and a series of state-of-the-art (SOTA) classification models. The experimental results demonstrated the superiority of our model for lung cancer subtypes classification with significant accuracy improvements in terms of accuracy (ACC), area under the curve (AUC), positive predictive value (PPV) and F1-score.

Synthesis of C-N Axially Chiral N-Arylbenzo[g]indoles via a Central-to-Axial Chirality Conversion Strategy.

Gold-catalyzed cascade cyclization of diynes for the synthesis of previously unexplored C-N axially chiral N-arylbenzo[g]indoles was described. The transformation was achieved via a central-to-axial chirality conversion strategy. The chiral conversion exhibited high efficiency. Besides single C-N chiral axis, N-arylbenzo[g]indoles bearing both C-N and C-C chiral axes were also afforded. The title compound derived monophosphine ligand was prepared and was evaluated in Pd-catalyzed asymmetric allylic substitutions, showing excellent chiral induction ability.

RAF1 facilitates KIT signaling and serves as a potential treatment target for gastrointestinal stromal tumor.

The aberrant activation of RAS/RAF/MEK/ERK signaling is important for KIT mutation-mediated tumorigenesis of gastrointestinal stromal tumor (GIST). In this study, we found that inhibition of RAF1 suppresses the activation of both wild-type KIT and primary KIT mutations in GIST, with primary KIT mutations showing greater sensitivity. This suggests a positive feedback loop between KIT and RAF1, wherein RAF1 facilitates KIT signaling. We further demonstrated that RAF1 associates with KIT and the kinase activity of RAF1 is necessary for its contribution to KIT activation. Accordingly, inhibition of RAF1 suppressed cell survival, proliferation, and cell cycle progression in vitro mediated by both wild-type KIT and primary KIT mutations. Inhibition of RAF1 in vivo suppressed GIST growth in a transgenic mouse model carrying germline KIT/V558A mutation, showing a similar treatment efficiency as imatinib, the first-line targeted therapeutic drug of GIST, while the combination use of imatinib and RAF1 inhibitor further suppressed tumor growth. Acquisition of drug-resistant secondary mutation of KIT is a major cause of treatment failure of GIST following targeted therapy. Like wild-type KIT and primary KIT mutations, inhibition of RAF1 suppressed the activation of secondary KIT mutation, and the cell survival, proliferation, cell cycle progression in vitro, and tumor growth in vivo mediated by secondary KIT mutation. However, the activation of secondary KIT mutation is less dependent on RAF1 compared with that of primary KIT mutations. Taken together, our results revealed that RAF1 facilitates KIT signaling and KIT mutation-mediated tumorigenesis of GIST, providing a rationale for further investigation into the use of RAF1 inhibitors alone or in combination with KIT inhibitor in the treatment of GIST, particularly in cases resistant to KIT inhibitors.

Four and a half LIM domains 2 (FHL2) attenuates tumorigenesis of gastrointestinal stromal tumors (GISTs) by negatively regulating KIT signaling.

Gastrointestinal stromal tumors (GISTs) are predominately induced by KIT mutants. In this study, we found that four and a half LIM domains 2 (FHL2) was highly expressed in GISTs and KIT signaling dramatically increased FHL2 transcription while FHL2 inhibited KIT transcription. In addition, our results showed that FHL2 associated with KIT and increased the ubiquitination of both wild-type KIT and primary KIT mutants in GISTs, leading to decreased expression and activation of KIT although primary KIT mutants were less inhibited by FHL2 than wild-type KIT. In the animal experiments, loss of FHL2 expression in mice carrying germline KIT/V558A mutation which can develop GISTs resulted in increased tumor growth, but increased sensitivity of GISTs to imatinib treatment which is used as the first-line targeted therapy of GISTs, suggesting that FHL2 plays a role in the response of GISTs to KIT inhibitor. Unlike wild-type KIT and primary KIT mutants, we further found that FHL2 didn't alter the expression and activation of drug-resistant secondary KIT mutants. Taken together, our results indicated that FHL2 acts as the negative feedback of KIT signaling in GISTs while primary KIT mutants are less sensitive and secondary KIT mutants are resistant to the inhibition of FHL2.

FPL+: Filtered Pseudo Label-based Unsupervised Cross-Modality Adaptation for 3D Medical Image Segmentation.

Adapting a medical image segmentation model to a new domain is important for improving its cross-domain transferability, and due to the expensive annotation process, Unsupervised Domain Adaptation (UDA) is appealing where only unlabeled images are needed for the adaptation. Existing UDA methods are mainly based on image or feature alignment with adversarial training for regularization, and they are limited by insufficient supervision in the target domain. In this paper, we propose an enhanced Filtered Pseudo Label (FPL+)-based UDA method for 3D medical image segmentation. It first uses cross-domain data augmentation to translate labeled images in the source domain to a dual-domain training set consisting of a pseudo source-domain set and a pseudo target-domain set. To leverage the dual-domain augmented images to train a pseudo label generator, domain-specific batch normalization layers are used to deal with the domain shift while learn the domain-invariant structure features, generating high-quality pseudo labels for target-domain images. We then combine labeled source-domain images and target-domain images with pseudo labels to train a final segmentor, where image-level weighting based on uncertainty estimation and pixel-level weighting based on dual-domain consensus are proposed to mitigate the adverse effect of noisy pseudo labels. Experiments on three public multi-modal datasets for Vestibular Schwannoma, brain tumor and whole heart segmentation show that our method surpassed ten state-of-the-art UDA methods, and it even achieved better results than fully supervised learning in the target domain in some cases.

Artificial intelligence-based assessment of PD-L1 expression in diffuse large B cell lymphoma.

Diffuse large B cell lymphoma (DLBCL) is an aggressive blood cancer known for its rapid progression and high incidence. The growing use of immunohistochemistry (IHC) has significantly contributed to the detailed cell characterization, thereby playing a crucial role in guiding treatment strategies for DLBCL. In this study, we developed an AI-based image analysis approach for assessing PD-L1 expression in DLBCL patients. PD-L1 expression represents as a major biomarker for screening patients who can benefit from targeted immunotherapy interventions. In particular, we performed large-scale cell annotations in IHC slides, encompassing over 5101 tissue regions and 146,439 live cells. Extensive experiments in primary and validation cohorts demonstrated the defined quantitative rule helped overcome the difficulty of identifying specific cell types. In assessing data obtained from fine needle biopsies, experiments revealed that there was a higher level of agreement in the quantitative results between Artificial Intelligence (AI) algorithms and pathologists, as well as among pathologists themselves, in comparison to the data obtained from surgical specimens. We highlight that the AI-enabled analytics enhance the objectivity and interpretability of PD-L1 quantification to improve the targeted immunotherapy development in DLBCL patients.

Effectiveness of general anaesthesia with remimazolam tosilate on intraoperative haemodynamics and postoperative recovery: study protocol for a randomised, positive-controlled, pragmatic clinical trial (GARTH trial).

INTRODUCTION: It is encouraged to estimate the effectiveness of components within the enhanced recovery after surgery (ERAS) protocol through patient-reported outcomes, alongside doctor-reported outcomes and length of hospital stay. At present, studies on the contributions of optimal anaesthetic drugs within the ERAS protocol to patient-reported and doctor-reported outcomes are limited. Therefore, this study aims to pragmatically evaluate the effectiveness and safety of general anaesthesia (GA) with remimazolam tosilate within the ERAS protocol on intraoperative haemodynamics and postoperative recovery in adults undergoing elective surgeries, compared with propofol. METHODS AND ANALYSIS: This study is a single-centre, randomised, blinded, positive-controlled, pragmatic clinical trial. A total of 900 patients, aged ≥18 years old, scheduled for an elective surgical procedure under GA will be included. Patients will be randomised in a 1:1 ratio to the remimazolam group (the GA with remimazolam tosilate within the ERAS protocol group) or propofol group (the GA with propofol within the ERAS protocol group), stratified by general surgery, thoracic surgery and other surgeries (including urological surgery and otolaryngology surgery). The primary outcomes include the 24-hour postoperative quality of recovery-40 score and the rate of intraoperative hypotension. Secondary endpoints include the rate of sedative hypotension requiring treatment, the haemodynamic profiles, the 72-hour postoperative quality of recovery-40 score, the functional anaesthetic capability, adverse events and complications, quality of life within 3 months as well as economic health outcomes. ETHICS AND DISSEMINATION: This study protocol has been approved by the ethics committee of Guangdong Provincial People's Hospital (KY-H-2022-005-03-08). Dissemination plans will be presented at scientific meetings and in scientific publications. TRIAL REGISTRATION NUMBER: ChiCTR2200062520.

Cell-permeable PI3 kinase competitive peptide inhibits KIT mutant mediated tumorigenesis of gastrointestinal stromal tumor (GIST).

BACKGROUND: Mutations in the receptor tyrosine kinase KIT are the main cause of gastrointestinal stromal tumor (GIST), and the KIT mutants mediated PI3 kinase activation plays a key role in the tumorigenesis of GIST. In this study, we aimed to block PI3 kinase activation by cell-permeable peptide and investigate its possible application in the treatment of GIST. METHODS AND RESULTS: We designed cell-permeable peptides based on the binding domain of PI3 kinase subunit p85 to KIT or PI3 kinase subunit p110, respectively, in order to compete for the binding between p85 and KIT or p110 and therefore inhibit the activation of PI3 kinases mediated by KIT. The results showed that the peptide can penetrate the cells, and inhibit the activation of PI3 kinases, leading to reduced cell survival and cell proliferation mediated by KIT mutants in vitro. Treatment of mice carrying germline KIT/V558A mutation, which can develop GIST, with the peptide that can compete for the binding between p85 and p110, led to reduced tumorigenesis of GIST. The peptide can further enhance the inhibition of the tumor growth by imatinib which is used as the first line targeted therapy of GIST. CONCLUSIONS: Our results showed that cell-permeable PI3 kinase competitive peptide can inhibit KIT-mediated PI3 kinase activation and tumorigenesis of GIST, providing a rationale to further test the peptide in the treatment of GIST and even other tumors with over-activation of PI3 kinases.

Pattern-Aware Transformer: Hierarchical Pattern Propagation in Sequential Medical Images.

This paper investigates how to effectively mine contextual information among sequential images and jointly model them in medical imaging tasks. Different from state-of-the-art methods that model sequential correlations via point-wise token encoding, this paper develops a novel hierarchical pattern-aware tokenization strategy. It handles distinct visual patterns independently and hierarchically, which not only ensures the full flexibility of attention aggregation under different pattern representations but also preserves both local and global information simultaneously. Based on this strategy, we propose a Pattern-Aware Transformer (PATrans) featuring a global-local dual-path pattern-aware cross-attention mechanism to achieve hierarchical pattern matching and propagation among sequential images. Furthermore, PATrans is plug-and-play and can be seamlessly integrated into various backbone networks for diverse downstream sequence modeling tasks. We demonstrate its general application paradigm across four domains and five benchmarks in video object detection and 3D volumetric semantic segmentation tasks, respectively. Impressively, PATrans sets new state-of-the-art across all these benchmarks, i.e., CVC-Video (92.3% detection F1), ASU-Mayo (99.1% localization F1), Lung Tumor (78.59% DSC), Nasopharynx Tumor (75.50% DSC), and Kidney Tumor (87.53% DSC). Codes and models are available at https://github.com/GGaoxiang/PATrans.

One-Shot Weakly-Supervised Segmentation in 3D Medical Images.

Deep neural networks typically require accurate and a large number of annotations to achieve outstanding performance in medical image segmentation. One-shot and weakly-supervised learning are promising research directions that reduce labeling effort by learning a new class from only one annotated image and using coarse labels instead, respectively. In this work, we present an innovative framework for 3D medical image segmentation with one-shot and weakly-supervised settings. Firstly a propagation-reconstruction network is proposed to propagate scribbles from one annotated volume to unlabeled 3D images based on the assumption that anatomical patterns in different human bodies are similar. Then a multi-level similarity denoising module is designed to refine the scribbles based on embeddings from anatomical- to pixel-level. After expanding the scribbles to pseudo masks, we observe the miss-classified voxels mainly occur at the border region and propose to extract self-support prototypes for the specific refinement. Based on these weakly-supervised segmentation results, we further train a segmentation model for the new class with the noisy label training strategy. Experiments on three CT and one MRI datasets show the proposed method obtains significant improvement over the state-of-the-art methods and performs robustly even under severe class imbalance and low contrast. Code is publicly available at https://github.com/LWHYC/OneShot_WeaklySeg.

Hierarchical-Instance Contrastive Learning for Minority Detection on Imbalanced Medical Datasets.

Deep learning methods are often hampered by issues such as data imbalance and data-hungry. In medical imaging, malignant or rare diseases are frequently of minority classes in the dataset, featured by diversified distribution. Besides that, insufficient labels and unseen cases also present conundrums for training on the minority classes. To confront the stated problems, we propose a novel Hierarchical-instance Contrastive Learning (HCLe) method for minority detection by only involving data from the majority class in the training stage. To tackle inconsistent intra-class distribution in majority classes, our method introduces two branches, where the first branch employs an auto-encoder network augmented with three constraint functions to effectively extract image-level features, and the second branch designs a novel contrastive learning network by taking into account the consistency of features among hierarchical samples from majority classes. The proposed method is further refined with a diverse mini-batch strategy, enabling the identification of minority classes under multiple conditions. Extensive experiments have been conducted to evaluate the proposed method on three datasets of different diseases and modalities. The experimental results show that the proposed method outperforms the state-of-the-art methods.