ScribSD+: Scribble-supervised medical image segmentation based on simultaneous multi-scale knowledge distillation and class-wise contrastive regularization.

Despite that deep learning has achieved state-of-the-art performance for automatic medical image segmentation, it often requires a large amount of pixel-level manual annotations for training. Obtaining these high-quality annotations is time-consuming and requires specialized knowledge, which hinders the widespread application that relies on such annotations to train a model with good segmentation performance. Using scribble annotations can substantially reduce the annotation cost, but often leads to poor segmentation performance due to insufficient supervision. In this work, we propose a novel framework named as ScribSD+ that is based on multi-scale knowledge distillation and class-wise contrastive regularization for learning from scribble annotations. For a student network supervised by scribbles and the teacher based on Exponential Moving Average (EMA), we first introduce multi-scale prediction-level Knowledge Distillation (KD) that leverages soft predictions of the teacher network to supervise the student at multiple scales, and then propose class-wise contrastive regularization which encourages feature similarity within the same class and dissimilarity across different classes, thereby effectively improving the segmentation performance of the student network. Experimental results on the ACDC dataset for heart structure segmentation and a fetal MRI dataset for placenta and fetal brain segmentation demonstrate that our method significantly improves the student's performance and outperforms five state-of-the-art scribble-supervised learning methods. Consequently, the method has a potential for reducing the annotation cost in developing deep learning models for clinical diagnosis.

Less confidence, less forgetting: Learning with a humbler teacher in exemplar-free Class-Incremental learning.

Class-Incremental learning (CIL) is challenging due to catastrophic forgetting (CF), which escalates in exemplar-free scenarios. To mitigate CF, Knowledge Distillation (KD), which leverages old models as teacher models, has been widely employed in CIL. However, based on a case study, our investigation reveals that the teacher model exhibits over-confidence in unseen new samples. In this article, we conduct empirical experiments and provide theoretical analysis to investigate the over-confident phenomenon and the impact of KD in exemplar-free CIL, where access to old samples is unavailable. Building on our analysis, we propose a novel approach, Learning with Humbler Teacher, by systematically selecting an appropriate checkpoint model as a humbler teacher to mitigate CF. Furthermore, we explore utilizing the nuclear norm to obtain an appropriate temporal ensemble to enhance model stability. Notably, LwHT outperforms the state-of-the-art approach by a significant margin of 10.41%, 6.56%, and 4.31% in various settings while demonstrating superior model plasticity.

Prediction of protein secondary structure by the improved TCN-BiLSTM-MHA model with knowledge distillation.

Secondary structure prediction is a key step in understanding protein function and biological properties and is highly important in the fields of new drug development, disease treatment, bioengineering, etc. Accurately predicting the secondary structure of proteins helps to reveal how proteins are folded and how they function in cells. The application of deep learning models in protein structure prediction is particularly important because of their ability to process complex sequence information and extract meaningful patterns and features, thus significantly improving the accuracy and efficiency of prediction. In this study, a combined model integrating an improved temporal convolutional network (TCN), bidirectional long short-term memory (BiLSTM), and a multi-head attention (MHA) mechanism is proposed to enhance the accuracy of protein prediction in both eight-state and three-state structures. One-hot encoding features and word vector representations of physicochemical properties are incorporated. A significant emphasis is placed on knowledge distillation techniques utilizing the ProtT5 pretrained model, leading to performance improvements. The improved TCN, achieved through multiscale fusion and bidirectional operations, allows for better extraction of amino acid sequence features than traditional TCN models. The model demonstrated excellent prediction performance on multiple datasets. For the TS115, CB513 and PDB (2018-2020) datasets, the prediction accuracy of the eight-state structure of the six datasets in this paper reached 88.2%, 84.9%, and 95.3%, respectively, and the prediction accuracy of the three-state structure reached 91.3%, 90.3%, and 96.8%, respectively. This study not only improves the accuracy of protein secondary structure prediction but also provides an important tool for understanding protein structure and function, which is particularly applicable to resource-constrained contexts and provides a valuable tool for understanding protein structure and function.

An Optimization Method for Lightweight Rock Classification Models: Transferred Rich Fine-Grained Knowledge.

Rock image classification represents a challenging fine-grained image classification task characterized by subtle differences among closely related rock categories. Current contrastive learning methods prevalently utilized in fine-grained image classification restrict the model's capacity to discern critical features contrastively from image pairs, and are typically too large for deployment on mobile devices used for in situ rock identification. In this work, we introduce an innovative and compact model generation framework anchored by the design of a Feature Positioning Comparison Network (FPCN). The FPCN facilitates interaction between feature vectors from localized regions within image pairs, capturing both shared and distinctive features. Further, it accommodates the variable scales of objects depicted in images, which correspond to differing quantities of inherent object information, directing the network's attention to additional contextual details based on object size variability. Leveraging knowledge distillation, the architecture is streamlined, with a focus on nuanced information at activation boundaries to master the precise fine-grained decision boundaries, thereby enhancing the small model's accuracy. Empirical evidence demonstrates that our proposed method based on FPCN improves the classification accuracy mobile lightweight models by nearly 2% while maintaining the same time and space consumption.

FedKG: A Knowledge Distillation-Based Federated Graph Method for Social Bot Detection.

Malicious social bots pose a serious threat to social network security by spreading false information and guiding bad opinions in social networks. The singularity and scarcity of single organization data and the high cost of labeling social bots have given rise to the construction of federated models that combine federated learning with social bot detection. In this paper, we first combine the federated learning framework with the Relational Graph Convolutional Neural Network (RGCN) model to achieve federated social bot detection. A class-level cross entropy loss function is applied in the local model training to mitigate the effects of the class imbalance problem in local data. To address the data heterogeneity issue from multiple participants, we optimize the classical federated learning algorithm by applying knowledge distillation methods. Specifically, we adjust the client-side and server-side models separately: training a global generator to generate pseudo-samples based on the local data distribution knowledge to correct the optimization direction of client-side classification models, and integrating client-side classification models' knowledge on the server side to guide the training of the global classification model. We conduct extensive experiments on widely used datasets, and the results demonstrate the effectiveness of our approach in social bot detection in heterogeneous data scenarios. Compared to baseline methods, our approach achieves a nearly 3-10% improvement in detection accuracy when the data heterogeneity is larger. Additionally, our method achieves the specified accuracy with minimal communication rounds.

Deep Transfer Learning Method Using Self-Pixel and Global Channel Attentive Regularization.

The purpose of this paper is to propose a novel transfer learning regularization method based on knowledge distillation. Recently, transfer learning methods have been used in various fields. However, problems such as knowledge loss still occur during the process of transfer learning to a new target dataset. To solve these problems, there are various regularization methods based on knowledge distillation techniques. In this paper, we propose a transfer learning regularization method based on feature map alignment used in the field of knowledge distillation. The proposed method is composed of two attention-based submodules: self-pixel attention (SPA) and global channel attention (GCA). The self-pixel attention submodule utilizes both the feature maps of the source and target models, so that it provides an opportunity to jointly consider the features of the target and the knowledge of the source. The global channel attention submodule determines the importance of channels through all layers, unlike the existing methods that calculate these only within a single layer. Accordingly, transfer learning regularization is performed by considering both the interior of each single layer and the depth of the entire layer. Consequently, the proposed method using both of these submodules showed overall improved classification accuracy than the existing methods in classification experiments on commonly used datasets.

NTCE-KD: Non-Target-Class-Enhanced Knowledge Distillation.

Most logit-based knowledge distillation methods transfer soft labels from the teacher model to the student model via Kullback-Leibler divergence based on softmax, an exponential normalization function. However, this exponential nature of softmax tends to prioritize the largest class (target class) while neglecting smaller ones (non-target classes), leading to an oversight of the non-target classes's significance. To address this issue, we propose Non-Target-Class-Enhanced Knowledge Distillation (NTCE-KD) to amplify the role of non-target classes both in terms of magnitude and diversity. Specifically, we present a magnitude-enhanced Kullback-Leibler (MKL) divergence multi-shrinking the target class to enhance the impact of non-target classes in terms of magnitude. Additionally, to enrich the diversity of non-target classes, we introduce a diversity-based data augmentation strategy (DDA), further enhancing overall performance. Extensive experimental results on the CIFAR-100 and ImageNet-1k datasets demonstrate that non-target classes are of great significance and that our method achieves state-of-the-art performance across a wide range of teacher-student pairs.

A novel fish individual recognition method for precision farming based on knowledge distillation strategy and the range of the receptive field.

With the continuous development of green and high-quality aquaculture technology, the process of industrialized aquaculture has been promoted. Automation, intelligence, and precision have become the future development trend of the aquaculture industry. Fish individual recognition can further distinguish fish individuals based on the determination of fish categories, providing basic support for fish disease analysis, bait feeding, and precision aquaculture. However, the high similarity of fish individuals and the complexity of the underwater environment presents great challenges to fish individual recognition. To address these problems, we propose a novel fish individual recognition method for precision farming that rethinks the knowledge distillation strategy and the chunking method in the vision transformer. The method uses the traditional convolutional neural network model as the teacher model, introducing the teacher token to guide the student model to learn the fish texture features. We propose stride patch embedding to expand the range of the receptive field, thus enhancing the local continuity of the image, and self-attention-pruning to discard unimportant tokens and reduce the model computation. The experimental results on the DlouFish dataset show that the proposed method in this paper improves accuracy by 3.25% compared to ECA Resnet152, with an accuracy of 93.19%, and also outperforms other vision transformer models.

Multistage feature fusion knowledge distillation.

Generally, the recognition performance of lightweight models is often lower than that of large models. Knowledge distillation, by teaching a student model using a teacher model, can further enhance the recognition accuracy of lightweight models. In this paper, we approach knowledge distillation from the perspective of intermediate feature-level knowledge distillation. We combine a cross-stage feature fusion symmetric framework, an attention mechanism to enhance the fused features, and a contrastive loss function for teacher and student models at the same stage to comprehensively implement a multistage feature fusion knowledge distillation method. This approach addresses the problem of significant differences in the intermediate feature distributions between teacher and student models, making it difficult to effectively learn implicit knowledge and thus improving the recognition accuracy of the student model. Compared to existing knowledge distillation methods, our method performs at a superior level. On the CIFAR100 dataset, it boosts the recognition accuracy of ResNet20 from 69.06% to 71.34%, and on the TinyImagenet dataset, it increases the recognition accuracy of ResNet18 from 66.54% to 68.03%, demonstrating the effectiveness and generalizability of our approach. Furthermore, there is room for further optimization of the overall distillation structure and feature extraction methods in this approach, which requires further research and exploration.

Self-architectural knowledge distillation for spiking neural networks.

Spiking neural networks (SNNs) have attracted attention due to their biological plausibility and the potential for low-energy applications on neuromorphic hardware. Two mainstream approaches are commonly used to obtain SNNs, i.e., ANN-to-SNN conversion methods, and Directly-trained-SNN methods. However, the former achieve excellent performance at the cost of a large number of time steps (i.e., latency), while the latter exhibit lower latency but suffers from suboptimal performance. To tackle the performance-latency trade-off, we propose Self-Architectural Knowledge Distillation (SAKD), an intuitive and effective method for SNNs leveraging Knowledge Distillation (KD). We adopt a bilevel teacher-student training strategy in SAKD, i.e., level-1 involves directly transferring same-architectural pre-trained ANN weights to SNNs, and level-2 encourages the SNNs to mimic ANN's behavior, considering both final responses and intermediate features aspects. Learning with informative supervision signals fostered by labels and ANNs, our SAKD achieves new state-of-the-art (SOTA) performance with a few time steps on widely-used classification benchmark datasets. On ImageNet-1K, with only 4 time steps, our Spiking-ResNet34 model attains a Top-1 accuracy of 70.04%, outperforming the previous same-architectural SOTA methods. Notably, our SEW-ResNet152 model reaches a Top-1 accuracy of 77.30% on ImageNet-1K, setting a new SOTA benchmark for SNNs. Furthermore, we apply our SAKD to various dense prediction downstream tasks, such as object detection and semantic segmentation, demonstrating strong generalization ability and superior performance. In conclusion, our proposed SAKD framework presents a promising approach for achieving both high performance and low latency in SNNs, potentially paving the way for future advancements in the field.

MV-MR: Multi-Views and Multi-Representations for Self-Supervised Learning and Knowledge Distillation.

We present a new method of self-supervised learning and knowledge distillation based on multi-views and multi-representations (MV-MR). MV-MR is based on the maximization of dependence between learnable embeddings from augmented and non-augmented views, jointly with the maximization of dependence between learnable embeddings from the augmented view and multiple non-learnable representations from the non-augmented view. We show that the proposed method can be used for efficient self-supervised classification and model-agnostic knowledge distillation. Unlike other self-supervised techniques, our approach does not use any contrastive learning, clustering, or stop gradients. MV-MR is a generic framework allowing the incorporation of constraints on the learnable embeddings via the usage of image multi-representations as regularizers. The proposed method is used for knowledge distillation. MV-MR provides state-of-the-art self-supervised performance on the STL10 and CIFAR20 datasets in a linear evaluation setup. We show that a low-complexity ResNet50 model pretrained using proposed knowledge distillation based on the CLIP ViT model achieves state-of-the-art performance on STL10 and CIFAR100 datasets.

G-MBRMD: Lightweight liver segmentation model based on guided teaching with multi-head boundary reconstruction mapping distillation.

BACKGROUND AND OBJECTIVES: Liver segmentation is pivotal for the quantitative analysis of liver cancer. Although current deep learning methods have garnered remarkable achievements for medical image segmentation, they come with high computational costs, significantly limiting their practical application in the medical field. Therefore, the development of an efficient and lightweight liver segmentation model becomes particularly important. METHODS: In our paper, we propose a real-time, lightweight liver segmentation model named G-MBRMD. Specifically, we employ a Transformer-based complex model as the teacher and a convolution-based lightweight model as the student. By introducing proposed multi-head mapping and boundary reconstruction strategies during the knowledge distillation process, Our method effectively guides the student model to gradually comprehend and master the global boundary processing capabilities of the complex teacher model, significantly enhancing the student model's segmentation performance without adding any computational complexity. RESULTS: On the LITS dataset, we conducted rigorous comparative and ablation experiments, four key metrics were used for evaluation, including model size, inference speed, Dice coefficient, and HD95. Compared to other methods, our proposed model achieved an average Dice coefficient of 90.14±16.78%, with only 0.6 MB memory and 0.095 s inference speed for a single image on a standard CPU. Importantly, this approach improved the average Dice coefficient of the baseline student model by 1.64% without increasing computational complexity. CONCLUSION: The results demonstrate that our method successfully realizes the unification of segmentation precision and lightness, and greatly enhances its potential for widespread application in practical settings.

Research on temperature detection method of liquor distilling pot feeding operation based on a compressed algorithm.

In the process of feeding the distilling bucket after vapor detection, the existing methods can only realize the lag detection after the steam overflow, and can not accurately detect the location of the steam, etc. At the same time, in order to effectively reduce the occupancy of the computational resources and improve the deployment performance, this study established infrared image dataset of fermented grains surface, and fused the YOLO v5n and the knowledge distillation and the model pruning algorithms, and an lightweight method YOLO v5ns-DP was proposed as as a model for detecting temperature changes in the surface layer of fermented grains during the process of feeding the distilling. The experimental results indicated that the improvement makes YOLOv5n improve its performance in all aspects. The number of parameters, GLOPs and model size of YOLO v5ns-DP have been reduced by 28.6%, 16.5%, and 26.4%, respectively, and the mAP has been improved by 0.6. Therefore, the algorithm is able to predict in advance and accurately detect the location of the liquor vapor, which effectively improves the precision and speed of the detection of the temperature of the surface fermented grains , and well completes the real-time detecting task.

Abdominal multi-organ segmentation in Multi-sequence MRIs based on visual attention guided network and knowledge distillation.

PURPOSE: The segmentation of abdominal organs in magnetic resonance imaging (MRI) plays a pivotal role in various therapeutic applications. Nevertheless, the application of deep-learning methods to abdominal organ segmentation encounters numerous challenges, especially in addressing blurred boundaries and regions characterized by low-contrast. METHODS: In this study, a multi-scale visual attention-guided network (VAG-Net) was proposed for abdominal multi-organ segmentation based on unpaired multi-sequence MRI. A new visual attention-guided (VAG) mechanism was designed to enhance the extraction of contextual information, particularly at the edge of organs. Furthermore, a new loss function inspired by knowledge distillation was introduced to minimize the semantic disparity between different MRI sequences. RESULTS: The proposed method was evaluated on the CHAOS 2019 Challenge dataset and compared with six state-of-the-art methods. The results demonstrated that our model outperformed these methods, achieving DSC values of 91.83 ± 0.24% and 94.09 ± 0.66% for abdominal multi-organ segmentation in T1-DUAL and T2-SPIR modality, respectively. CONCLUSION: The experimental results show that our proposed method has superior performance in abdominal multi-organ segmentation, especially in the case of small organs such as the kidneys.

Distillation of multi-class cervical lesion cell detection via synthesis-aided pre-training and patch-level feature alignment.

Automated detection of cervical abnormal cells from Thin-prep cytologic test (TCT) images is crucial for efficient cervical abnormal screening using computer-aided diagnosis systems. However, the construction of the detection model is hindered by the preparation of the training images, which usually suffers from issues of class imbalance and incomplete annotations. Additionally, existing methods often overlook the visual feature correlations among cells, which are crucial in cervical lesion cell detection as pathologists commonly rely on surrounding cells for identification. In this paper, we propose a distillation framework that utilizes a patch-level pre-training network to guide the training of an image-level detection network, which can be applied to various detectors without changing their architectures during inference. The main contribution is three-fold: (1) We propose the Balanced Pre-training Model (BPM) as the patch-level cervical cell classification model, which employs an image synthesis model to construct a class-balanced patch dataset for pre-training. (2) We design the Score Correction Loss (SCL) to enable the detection network to distill knowledge from the BPM model, thereby mitigating the impact of incomplete annotations. (3) We design the Patch Correlation Consistency (PCC) strategy to exploit the correlation information of extracted cells, consistent with the behavior of cytopathologists. Experiments on public and private datasets demonstrate the superior performance of the proposed distillation method, as well as its adaptability to various detection architectures.

AdaDFKD: Exploring adaptive inter-sample relationship in data-free knowledge distillation.

In scenarios like privacy protection or large-scale data transmission, data-free knowledge distillation (DFKD) methods are proposed to learn Knowledge Distillation (KD) when data is not accessible. They generate pseudo samples by extracting the knowledge from teacher model, and utilize above pseudo samples for KD. The challenge in previous DFKD methods lies in the static nature of their target distributions and they focus on learning the instance-level distributions, causing its reliance on the pretrained teacher model. To address above concerns, our study introduces a novel DFKD approach known as AdaDFKD, designed to establish and utilize relationships among pseudo samples, which is adaptive to the student model, and finally effectively mitigates the aforementioned risk. We achieve this by generating from "easy-to-discriminate" samples to "hard-to-discriminate" samples as human does. We design a relationship refinement module (R2M) to optimize the generation process, wherein we learn a progressive conditional distribution of negative samples and maximize the log-likelihood of inter-sample similarity of pseudosamples. Theoretically, we discover that such design of AdaDFKD both minimize the divergence and maximize the mutual information between the distribution of teacher and student models. Above results demonstrate the superiority of our approach over state-of-the-art (SOTA) DFKD methods across various benchmarks, teacher-student pairs, and evaluation metrics, as well as robustness and fast convergence.

A novel garment transfer method supervised by distilled knowledge of virtual try-on model.

Garment transfer can wear the garment of the model image onto the personal image. As garment transfer leverages wild and cheap garment input, it has attracted tremendous attention in the community and has a huge commercial potential. Since the ground truth of garment transfer is almost unavailable in reality, previous studies have treated garment transfer as either pose transfer or garment-pose disentanglement, and trained garment transfer in self-supervised learning, However, these implementation methods do not cover garment transfer intentions completely and face the robustness issue in the testing phase. Notably, virtual try-on technology has exhibited superior performance using self-supervised learning, we propose to supervise the garment transfer training via knowledge distillation from virtual try-on. Specifically, the overall pipeline is first to infer a garment transfer parsing, and to use it to guide downstream warping and inpainting tasks. The transfer parsing reasoning model learns the response and feature knowledge from the try-on parsing reasoning model and absorbs the hard knowledge from the ground truth. The progressive flow warping model learns the content knowledge from virtual try-on for a reasonable and precise garment warping. To enhance transfer realism, we propose an arm regrowth task to infer exposed skin. Experiments demonstrate that our method has state-of-the-art performance in transferring garments between persons compared with other virtual try-on and garment transfer methods.

Prototype-based sample-weighted distillation unified framework adapted to missing modality sentiment analysis.

Missing modality sentiment analysis is a prevalent and challenging issue in real life. Furthermore, the heterogeneity of multimodality often leads to an imbalance in optimization when attempting to optimize the same objective across all modalities in multimodal networks. Previous works have consistently overlooked the optimization imbalance of the network in cases when modalities are absent. This paper presents a Prototype-Based Sample-Weighted Distillation Unified Framework Adapted to Missing Modality Sentiment Analysis (PSWD). Specifically, it fuses features with a more efficient transformer-based cross-modal hierarchical cyclic fusion module. Subsequently, we propose two strategies, namely sample-weighted distillation and prototype regularization network, to address the issues of missing modality and optimization imbalance. The sample-weighted distillation strategy assigns higher weights to samples that are located closer to class boundaries. This facilitates the obtaining of complete knowledge by the student network from the teacher's network. The prototype regularization network calculates a balanced metric for each modality, which adaptively adjusts the gradient based on the prototype cross-entropy loss. Unlike conventional approaches, PSWD not only connects the sentiment analysis study in the missing modality to the full modality, but the proposed prototype regularization network is not reliant on the network structure and can be expanded to more multimodal studies. Massive experiments conducted on IEMOCAP and MSP-IMPROV show that our method achieves the best results compared to the latest baseline methods, which demonstrates its value for application in sentiment analysis.

Efficient skin lesion segmentation with boundary distillation.

Medical image segmentation models are commonly known for their complex structures, which often render them impractical for use on edge computing devices and compromising efficiency in the segmentation process. In light of this, the industry has proposed the adoption of knowledge distillation techniques. Nevertheless, the vast majority of existing knowledge distillation methods are focused on the classification tasks of skin diseases. Specifically, for the segmentation tasks of dermoscopy lesion images, these knowledge distillation methods fail to fully recognize the importance of features in the boundary regions of lesions within medical images, lacking boundary awareness for skin lesions. This paper introduces pioneering medical image knowledge distillation architecture. The aim of this method is to facilitate the efficient transfer of knowledge from existing complex medical image segmentation networks to a more simplified student network. Initially, a masked boundary feature (MBF) distillation module is designed. By applying random masking to the periphery of skin lesions, the MBF distillation module obliges the student network to reproduce the comprehensive features of the teacher network. This process, in turn, augments the representational capabilities of the student network. Building on the MBF distillation module, this paper employs a cascaded combination approach to integrate the MBF distillation module into a multi-head boundary feature (M2BF) distillation module, further strengthening the student network's feature learning ability and enhancing the overall image segmentation performance of the distillation model. This method has been experimentally validated on the public datasets ISIC-2016 and PH2, with results showing significant performance improvements in the student network. Our findings highlight the practical utility of the lightweight network distilled using our approach, particularly in scenarios demanding high operational speed and minimal storage usage. This research offers promising prospects for practical applications in the realm of medical image segmentation.

Global prototype distillation for heterogeneous federated learning.

Federated learning is a distributed machine learning paradigm where the goal is to collaboratively train a high quality global model while private training data remains local over distributed clients. However, heterogenous data distribution over clients is severely challenging for federated learning system, which severely damage the quality of model. In order to address this challenge, we propose global prototype distillation (FedGPD) for heterogenous federated learning to improve performance of global model. The intuition is to use global class prototypes as knowledge to instruct local training on client side. Eventually, local objectives will be consistent with the global optima so that FedGPD learns an improved global model. Experiments show that FedGPD outperforms previous state-of-art methods by 0.22% ~1.28% in terms of average accuracy on representative benchmark datasets.