Data-Free Adversarial Distillation
  Knowledge Distillation (KD) has made remarkable progress in the last few
years and become a popular paradigm for model compression and knowledge
transfer. However, almost all existing KD algorithms are data-driven, i.e.,
relying on a large amount of original training data or alternative data, which
is usually unavailable in real-world scenarios. In this paper, we devote
ourselves to this challenging problem and propose a novel adversarial
distillation mechanism to craft a compact student model without any real-world
data. We introduce a model discrepancy to quantificationally measure the
difference between student and teacher models and construct an optimizable
upper bound. In our work, the student and the teacher jointly act the role of
the discriminator to reduce this discrepancy, when a generator adversarially
produces some "hard samples" to enlarge it. Extensive experiments demonstrate
that the proposed data-free method yields comparable performance to existing
data-driven methods. More strikingly, our approach can be directly extended to
semantic segmentation, which is more complicated than classification, and our
approach achieves state-of-the-art results. Code and pretrained models are
available at https://github.com/VainF/Data-Free-Adversarial-Distillation.
