Mitigating Memorization of Noisy Labels via Regularization between
  Representations
  Designing robust loss functions is popular in learning with noisy labels
while existing designs did not explicitly consider the overfitting property of
deep neural networks (DNNs). As a result, applying these losses may still
suffer from overfitting/memorizing noisy labels as training proceeds. In this
paper, we first theoretically analyze the memorization effect and show that a
lower-capacity model may perform better on noisy datasets. However, it is
non-trivial to design a neural network with the best capacity given an
arbitrary task. To circumvent this dilemma, instead of changing the model
architecture, we decouple DNNs into an encoder followed by a linear classifier
and propose to restrict the function space of a DNN by a representation
regularizer. Particularly, we require the distance between two self-supervised
features to be positively related to the distance between the corresponding two
supervised model outputs. Our proposed framework is easily extendable and can
incorporate many other robust loss functions to further improve performance.
Extensive experiments and theoretical analyses support our claims. Code is
available at github.com/UCSC-REAL/SelfSup_NoisyLabel.
