P-KDGAN: Progressive Knowledge Distillation with GANs for One-class
  Novelty Detection
  One-class novelty detection is to identify anomalous instances that do not
conform to the expected normal instances. In this paper, the Generative
Adversarial Networks (GANs) based on encoder-decoder-encoder pipeline are used
for detection and achieve state-of-the-art performance. However, deep neural
networks are too over-parameterized to deploy on resource-limited devices.
Therefore, Progressive Knowledge Distillation with GANs (PKDGAN) is proposed to
learn compact and fast novelty detection networks. The P-KDGAN is a novel
attempt to connect two standard GANs by the designed distillation loss for
transferring knowledge from the teacher to the student. The progressive
learning of knowledge distillation is a two-step approach that continuously
improves the performance of the student GAN and achieves better performance
than single step methods. In the first step, the student GAN learns the basic
knowledge totally from the teacher via guiding of the pretrained teacher GAN
with fixed weights. In the second step, joint fine-training is adopted for the
knowledgeable teacher and student GANs to further improve the performance and
stability. The experimental results on CIFAR-10, MNIST, and FMNIST show that
our method improves the performance of the student GAN by 2.44%, 1.77%, and
1.73% when compressing the computation at ratios of 24.45:1, 311.11:1, and
700:1, respectively.
