Zero-shot Knowledge Transfer via Adversarial Belief Matching
  Performing knowledge transfer from a large teacher network to a smaller
student is a popular task in modern deep learning applications. However, due to
growing dataset sizes and stricter privacy regulations, it is increasingly
common not to have access to the data that was used to train the teacher. We
propose a novel method which trains a student to match the predictions of its
teacher without using any data or metadata. We achieve this by training an
adversarial generator to search for images on which the student poorly matches
the teacher, and then using them to train the student. Our resulting student
closely approximates its teacher for simple datasets like SVHN, and on CIFAR10
we improve on the state-of-the-art for few-shot distillation (with 100 images
per class), despite using no data. Finally, we also propose a metric to
quantify the degree of belief matching between teacher and student in the
vicinity of decision boundaries, and observe a significantly higher match
between our zero-shot student and the teacher, than between a student distilled
with real data and the teacher. Code available at:
https://github.com/polo5/ZeroShotKnowledgeTransfer
