CoDC: Accurate Learning with Noisy Labels via Disagreement and Consistency.

Inspired by the biological nervous system, deep neural networks (DNNs) are able to achieve remarkable performance in various tasks. However, they struggle to handle label noise, which can poison the memorization effects of DNNs. Co-teaching-based methods are popular in learning with noisy labels. These methods cross-train two DNNs based on the small-loss criterion and employ a strategy using either "disagreement" or "consistency" to obtain the divergence of the two networks. However, these methods are sample-inefficient for generalization in noisy scenarios. In this paper, we propose CoDC, a novel Co-teaching-basedmethod for accurate learning with label noise via both Disagreement and Consistency strategies. Specifically, CoDC maintains disagreement at the feature level and consistency at the prediction level using a balanced loss function. Additionally, a weighted cross-entropy loss is proposed based on information derived from the historical training process. Moreover, the valuable knowledge involved in "large-loss" samples is further developed and utilized by assigning pseudo-labels. Comprehensive experiments were conducted on both synthetic and real-world noise and under various noise types. CoDC achieved 72.81% accuracy on the Clothing1M dataset and 76.96% (Top1) accuracy on the WebVision1.0 dataset. These superior results demonstrate the effectiveness and robustness of learning with noisy labels.

Meta-probability Weighting for Improving Reliability of DNNs to Label Noise.

Training noise-robust deep neural networks (DNNs) in label noise scenario is a crucial task. In this paper, we first demonstrates that the DNNs learning with label noise exhibits over-fitting issue on noisy labels because of the DNNs is too confidence in its learning capacity. More significantly, however, it also potentially suffers from under-learning on samples with clean labels. DNNs essentially should pay more attention on the clean samples rather than the noisy samples. Inspired by the sample-weighting strategy, we propose a meta-probability weighting (MPW) algorithm which weights the output probability of DNNs to prevent DNNs from over-fitting to label noise and alleviate the under-learning issue on the clean sample. MPW conducts an approximation optimization to adaptively learn the probability weights from data under the supervision of a small clean dataset, and achieves iterative optimization between probability weights and network parameters via meta-learning paradigm. The ablation studies substantiate the effectiveness of MPW to prevent the deep neural networks from overfitting to label noise and improve the learning capacity on clean samples. Furthermore, MPW achieves competitive performance with other state-of-the-art methods on both synthetic and real-world noises.

Deep learning methods may not outperform other machine learning methods on analyzing genomic studies.

Deep Learning (DL) has been broadly applied to solve big data problems in biomedical fields, which is most successful in image processing. Recently, many DL methods have been applied to analyze genomic studies. However, genomic data usually has too small a sample size to fit a complex network. They do not have common structural patterns like images to utilize pre-trained networks or take advantage of convolution layers. The concern of overusing DL methods motivates us to evaluate DL methods' performance versus popular non-deep Machine Learning (ML) methods for analyzing genomic data with a wide range of sample sizes. In this paper, we conduct a benchmark study using the UK Biobank data and its many random subsets with different sample sizes. The original UK Biobank data has about 500k participants. Each patient has comprehensive patient characteristics, disease histories, and genomic information, i.e., the genotypes of millions of Single-Nucleotide Polymorphism (SNPs). We are interested in predicting the risk of three lung diseases: asthma, COPD, and lung cancer. There are 205,238 participants have recorded disease outcomes for these three diseases. Five prediction models are investigated in this benchmark study, including three non-deep machine learning methods (Elastic Net, XGBoost, and SVM) and two deep learning methods (DNN and LSTM). Besides the most popular performance metrics, such as the F1-score, we promote the hit curve, a visual tool to describe the performance of predicting rare events. We discovered that DL methods frequently fail to outperform non-deep ML in analyzing genomic data, even in large datasets with over 200k samples. The experiment results suggest not overusing DL methods in genomic studies, even with biobank-level sample sizes. The performance differences between DL and non-deep ML decrease as the sample size of data increases. This suggests when the sample size of data is significant, further increasing sample sizes leads to more performance gain in DL methods. Hence, DL methods could be better if we analyze genomic data bigger than this study.