On the Convergence of A Family of Robust Losses for Stochastic Gradient
  Descent
  The convergence of Stochastic Gradient Descent (SGD) using convex loss
functions has been widely studied. However, vanilla SGD methods using convex
losses cannot perform well with noisy labels, which adversely affect the update
of the primal variable in SGD methods. Unfortunately, noisy labels are
ubiquitous in real world applications such as crowdsourcing. To handle noisy
labels, in this paper, we present a family of robust losses for SGD methods. By
employing our robust losses, SGD methods successfully reduce negative effects
caused by noisy labels on each update of the primal variable. We not only
reveal that the convergence rate is O(1/T) for SGD methods using robust losses,
but also provide the robustness analysis on two representative robust losses.
Comprehensive experimental results on six real-world datasets show that SGD
methods using robust losses are obviously more robust than other baseline
methods in most situations with fast convergence.
