Data-Free Network Quantization With Adversarial Knowledge Distillation
  Network quantization is an essential procedure in deep learning for
development of efficient fixed-point inference models on mobile or edge
platforms. However, as datasets grow larger and privacy regulations become
stricter, data sharing for model compression gets more difficult and
restricted. In this paper, we consider data-free network quantization with
synthetic data. The synthetic data are generated from a generator, while no
data are used in training the generator and in quantization. To this end, we
propose data-free adversarial knowledge distillation, which minimizes the
maximum distance between the outputs of the teacher and the (quantized) student
for any adversarial samples from a generator. To generate adversarial samples
similar to the original data, we additionally propose matching statistics from
the batch normalization layers for generated data and the original data in the
teacher. Furthermore, we show the gain of producing diverse adversarial samples
by using multiple generators and multiple students. Our experiments show the
state-of-the-art data-free model compression and quantization results for
(wide) residual networks and MobileNet on SVHN, CIFAR-10, CIFAR-100, and
Tiny-ImageNet datasets. The accuracy losses compared to using the original
datasets are shown to be very minimal.
