Dataset Distillation
  Model distillation aims to distill the knowledge of a complex model into a
simpler one. In this paper, we consider an alternative formulation called
dataset distillation: we keep the model fixed and instead attempt to distill
the knowledge from a large training dataset into a small one. The idea is to
synthesize a small number of data points that do not need to come from the
correct data distribution, but will, when given to the learning algorithm as
training data, approximate the model trained on the original data. For example,
we show that it is possible to compress 60,000 MNIST training images into just
10 synthetic distilled images (one per class) and achieve close to original
performance with only a few gradient descent steps, given a fixed network
initialization. We evaluate our method in various initialization settings and
with different learning objectives. Experiments on multiple datasets show the
advantage of our approach compared to alternative methods.
