AutoDO: Robust AutoAugment for Biased Data with Label Noise via Scalable
  Probabilistic Implicit Differentiation
  AutoAugment has sparked an interest in automated augmentation methods for
deep learning models. These methods estimate image transformation policies for
train data that improve generalization to test data. While recent papers
evolved in the direction of decreasing policy search complexity, we show that
those methods are not robust when applied to biased and noisy data. To overcome
these limitations, we reformulate AutoAugment as a generalized automated
dataset optimization (AutoDO) task that minimizes the distribution shift
between test data and distorted train dataset. In our AutoDO model, we
explicitly estimate a set of per-point hyperparameters to flexibly change
distribution of train data. In particular, we include hyperparameters for
augmentation, loss weights, and soft-labels that are jointly estimated using
implicit differentiation. We develop a theoretical probabilistic interpretation
of this framework using Fisher information and show that its complexity scales
linearly with the dataset size. Our experiments on SVHN, CIFAR-10/100, and
ImageNet classification show up to 9.3% improvement for biased datasets with
label noise compared to prior methods and, importantly, up to 36.6% gain for
underrepresented SVHN classes.
