Active Bias: Training More Accurate Neural Networks by Emphasizing High
  Variance Samples
  Self-paced learning and hard example mining re-weight training instances to
improve learning accuracy. This paper presents two improved alternatives based
on lightweight estimates of sample uncertainty in stochastic gradient descent
(SGD): the variance in predicted probability of the correct class across
iterations of mini-batch SGD, and the proximity of the correct class
probability to the decision threshold. Extensive experimental results on six
datasets show that our methods reliably improve accuracy in various network
architectures, including additional gains on top of other popular training
techniques, such as residual learning, momentum, ADAM, batch normalization,
dropout, and distillation.
