Data-Free Knowledge Distillation for Deep Neural Networks
  Recent advances in model compression have provided procedures for compressing
large neural networks to a fraction of their original size while retaining most
if not all of their accuracy. However, all of these approaches rely on access
to the original training set, which might not always be possible if the network
to be compressed was trained on a very large dataset, or on a dataset whose
release poses privacy or safety concerns as may be the case for biometrics
tasks. We present a method for data-free knowledge distillation, which is able
to compress deep neural networks trained on large-scale datasets to a fraction
of their size leveraging only some extra metadata to be provided with a
pretrained model release. We also explore different kinds of metadata that can
be used with our method, and discuss tradeoffs involved in using each of them.
