Teacher-Student Compression with Generative Adversarial Networks
  More accurate machine learning models often demand more computation and
memory at test time, making them difficult to deploy on CPU- or
memory-constrained devices. Teacher-student compression (TSC), also known as
distillation, alleviates this burden by training a less expensive student model
to mimic the expensive teacher model while maintaining most of the original
accuracy. However, when fresh data is unavailable for the compression task, the
teacher's training data is typically reused, leading to suboptimal compression.
In this work, we propose to augment the compression dataset with synthetic data
from a generative adversarial network (GAN) designed to approximate the
training data distribution. Our GAN-assisted TSC (GAN-TSC) significantly
improves student accuracy for expensive models such as large random forests and
deep neural networks on both tabular and image datasets. Building on these
results, we propose a comprehensive metric---the TSC Score---to evaluate the
quality of synthetic datasets based on their induced TSC performance. The TSC
Score captures both data diversity and class affinity, and we illustrate its
benefits over the popular Inception Score in the context of image
classification.
