VICReg: Variance-Invariance-Covariance Regularization For Self-Supervised Learning

International audienceRecent self-supervised methods for image representation learning maximize theagreement between embedding vectors produced by encoders fed with differentviews of the same image. The main challenge is to prevent a collapse in whichthe encoders produce constant or non-informative vectors. We introduce VICReg(Variance-Invariance-Covariance Regularization), a method that explicitly avoidsthe collapse problem with two regularizations terms applied to both embeddingsseparately: (1) a term that maintains the variance of each embedding dimensionabove a threshold, (2) a term that decorrelates each pair of variables. Unlikemost other approaches to the same problem, VICReg does not require techniquessuch as: weight sharing between the branches, batch normalization, feature-wisenormalization, output quantization, stop gradient, memory banks, etc., and achievesresults on par with the state of the art on several downstream tasks. In addition, weshow that our variance regularization term stabilizes the training of other methodsand leads to performance improvements