Curriculum Loss: Robust Learning and Generalization against Label
  Corruption
  Deep neural networks (DNNs) have great expressive power, which can even
memorize samples with wrong labels. It is vitally important to reiterate
robustness and generalization in DNNs against label corruption. To this end,
this paper studies the 0-1 loss, which has a monotonic relationship with an
empirical adversary (reweighted) risk~\citep{hu2016does}. Although the 0-1 loss
has some robust properties, it is difficult to optimize. To efficiently
optimize the 0-1 loss while keeping its robust properties, we propose a very
simple and efficient loss, i.e. curriculum loss (CL). Our CL is a tighter upper
bound of the 0-1 loss compared with conventional summation based surrogate
losses. Moreover, CL can adaptively select samples for model training. As a
result, our loss can be deemed as a novel perspective of curriculum sample
selection strategy, which bridges a connection between curriculum learning and
robust learning. Experimental results on benchmark datasets validate the
robustness of the proposed loss.
