ProSelfLC: Progressive Self Label Correction for Training Robust Deep
  Neural Networks
  To train robust deep neural networks (DNNs), we systematically study several
target modification approaches, which include output regularisation, self and
non-self label correction (LC). Two key issues are discovered: (1) Self LC is
the most appealing as it exploits its own knowledge and requires no extra
models. However, how to automatically decide the trust degree of a learner as
training goes is not well answered in the literature? (2) Some methods penalise
while the others reward low-entropy predictions, prompting us to ask which one
is better?
  To resolve the first issue, taking two well-accepted propositions--deep
neural networks learn meaningful patterns before fitting noise [3] and minimum
entropy regularisation principle [10]--we propose a novel end-to-end method
named ProSelfLC, which is designed according to learning time and entropy.
Specifically, given a data point, we progressively increase trust in its
predicted label distribution versus its annotated one if a model has been
trained for enough time and the prediction is of low entropy (high confidence).
For the second issue, according to ProSelfLC, we empirically prove that it is
better to redefine a meaningful low-entropy status and optimise the learner
toward it. This serves as a defence of entropy minimisation.
  We demonstrate the effectiveness of ProSelfLC through extensive experiments
in both clean and noisy settings. The source code is available at
https://github.com/XinshaoAmosWang/ProSelfLC-CVPR2021.
  Keywords: entropy minimisation, maximum entropy, confidence penalty, self
knowledge distillation, label correction, label noise, semi-supervised
learning, output regularisation
