Learning with Bad Training Data via Iterative Trimmed Loss Minimization
  In this paper, we study a simple and generic framework to tackle the problem
of learning model parameters when a fraction of the training samples are
corrupted. We first make a simple observation: in a variety of such settings,
the evolution of training accuracy (as a function of training epochs) is
different for clean and bad samples. Based on this we propose to iteratively
minimize the trimmed loss, by alternating between (a) selecting samples with
lowest current loss, and (b) retraining a model on only these samples. We prove
that this process recovers the ground truth (with linear convergence rate) in
generalized linear models with standard statistical assumptions.
Experimentally, we demonstrate its effectiveness in three settings: (a) deep
image classifiers with errors only in labels, (b) generative adversarial
networks with bad training images, and (c) deep image classifiers with
adversarial (image, label) pairs (i.e., backdoor attacks). For the well-studied
setting of random label noise, our algorithm achieves state-of-the-art
performance without having access to any a-priori guaranteed clean samples.
