Understanding Whitening Loss in Self-supervised Learning.
IEEE Trans Pattern Anal Mach Intell
; PP2024 Jun 21.
Article
in En
| MEDLINE
| ID: mdl-38905085
ABSTRACT
A desirable objective in self-supervised learning (SSL) is to avoid feature collapse. Whitening loss guarantees collapse avoidance by minimizing the distance between embeddings of positive pairs under the conditioning that the embeddings from different views are whitened. In this paper, we propose a framework with an informative indicator to analyze whitening loss, which provides a clue to demystify several interesting phenomena and a pivoting point connecting to other SSL methods. We show that batch whitening (BW) based methods do not impose whitening constraints on the embedding but only require the embedding to be full-rank. This full-rank constraint is also sufficient to avoid dimensional collapse. We further demonstrate that the stable rank of the embedding is invariant during training by gradient descent, given the assumption that embedding is updated with an infinitely small learning rate. Based on our analysis, we propose channel whitening with random group partition (CW-RGP), which exploits the advantages of BW-based methods in preventing collapse and avoids their disadvantages requiring large batch size. Experimental results on ImageNet classification and COCO object detection reveal that the proposed CW-RGP possesses a promising potential for learning good representations.
Full text:
1
Collection:
01-internacional
Database:
MEDLINE
Language:
En
Journal:
IEEE Trans Pattern Anal Mach Intell
Journal subject:
INFORMATICA MEDICA
Year:
2024
Document type:
Article
Country of publication:
Estados Unidos