Co-teaching: Robust Training of Deep Neural Networks with Extremely
  Noisy Labels
  Deep learning with noisy labels is practically challenging, as the capacity
of deep models is so high that they can totally memorize these noisy labels
sooner or later during training. Nonetheless, recent studies on the
memorization effects of deep neural networks show that they would first
memorize training data of clean labels and then those of noisy labels.
Therefore in this paper, we propose a new deep learning paradigm called
Co-teaching for combating with noisy labels. Namely, we train two deep neural
networks simultaneously, and let them teach each other given every mini-batch:
firstly, each network feeds forward all data and selects some data of possibly
clean labels; secondly, two networks communicate with each other what data in
this mini-batch should be used for training; finally, each network back
propagates the data selected by its peer network and updates itself. Empirical
results on noisy versions of MNIST, CIFAR-10 and CIFAR-100 demonstrate that
Co-teaching is much superior to the state-of-the-art methods in the robustness
of trained deep models.
