To Smooth or Not? When Label Smoothing Meets Noisy Labels
  Label smoothing (LS) is an arising learning paradigm that uses the positively
weighted average of both the hard training labels and uniformly distributed
soft labels. It was shown that LS serves as a regularizer for training data
with hard labels and therefore improves the generalization of the model. Later
it was reported LS even helps with improving robustness when learning with
noisy labels. However, we observed that the advantage of LS vanishes when we
operate in a high label noise regime. Intuitively speaking, this is due to the
increased entropy of $\mathbb{P}(\text{noisy label}|X)$ when the noise rate is
high, in which case, further applying LS tends to "oversmooth" the estimated
posterior. We proceeded to discover that several learning-with-noisy-labels
solutions in the literature instead relate more closely to negative/not label
smoothing (NLS), which acts counter to LS and defines as using a negative
weight to combine the hard and soft labels! We provide understandings for the
properties of LS and NLS when learning with noisy labels. Among other
established properties, we theoretically show NLS is considered more beneficial
when the label noise rates are high. We provide extensive experimental results
on multiple benchmarks to support our findings too.
