Towards Understanding Label Smoothing
  Label smoothing regularization (LSR) has a great success in training deep
neural networks by stochastic algorithms such as stochastic gradient descent
and its variants. However, the theoretical understanding of its power from the
view of optimization is still rare. This study opens the door to a deep
understanding of LSR by initiating the analysis. In this paper, we analyze the
convergence behaviors of stochastic gradient descent with label smoothing
regularization for solving non-convex problems and show that an appropriate LSR
can help to speed up the convergence by reducing the variance. More
interestingly, we proposed a simple yet effective strategy, namely Two-Stage
LAbel smoothing algorithm (TSLA), that uses LSR in the early training epochs
and drops it off in the later training epochs. We observe from the improved
convergence result of TSLA that it benefits from LSR in the first stage and
essentially converges faster in the second stage. To the best of our knowledge,
this is the first work for understanding the power of LSR via establishing
convergence complexity of stochastic methods with LSR in non-convex
optimization. We empirically demonstrate the effectiveness of the proposed
method in comparison with baselines on training ResNet models over benchmark
data sets.
