SIGUA: Forgetting May Make Learning with Noisy Labels More Robust
  Given data with noisy labels, over-parameterized deep networks can gradually
memorize the data, and fit everything in the end. Although equipped with
corrections for noisy labels, many learning methods in this area still suffer
overfitting due to undesired memorization. In this paper, to relieve this
issue, we propose stochastic integrated gradient underweighted ascent (SIGUA):
in a mini-batch, we adopt gradient descent on good data as usual, and
learning-rate-reduced gradient ascent on bad data; the proposal is a versatile
approach where data goodness or badness is w.r.t. desired or undesired
memorization given a base learning method. Technically, SIGUA pulls
optimization back for generalization when their goals conflict with each other;
philosophically, SIGUA shows forgetting undesired memorization can reinforce
desired memorization. Experiments demonstrate that SIGUA successfully
robustifies two typical base learning methods, so that their performance is
often significantly improved.
