Why resampling outperforms reweighting for correcting sampling bias with
  stochastic gradients
  A data set sampled from a certain population is biased if the subgroups of
the population are sampled at proportions that are significantly different from
their underlying proportions. Training machine learning models on biased data
sets requires correction techniques to compensate for the bias. We consider two
commonly-used techniques, resampling and reweighting, that rebalance the
proportions of the subgroups to maintain the desired objective function. Though
statistically equivalent, it has been observed that resampling outperforms
reweighting when combined with stochastic gradient algorithms. By analyzing
illustrative examples, we explain the reason behind this phenomenon using tools
from dynamical stability and stochastic asymptotics. We also present
experiments from regression, classification, and off-policy prediction to
demonstrate that this is a general phenomenon. We argue that it is imperative
to consider the objective function design and the optimization algorithm
together while addressing the sampling bias.
