Scalable Model Compression by Entropy Penalized Reparameterization
  We describe a simple and general neural network weight compression approach,
in which the network parameters (weights and biases) are represented in a
"latent" space, amounting to a reparameterization. This space is equipped with
a learned probability model, which is used to impose an entropy penalty on the
parameter representation during training, and to compress the representation
using a simple arithmetic coder after training. Classification accuracy and
model compressibility is maximized jointly, with the bitrate--accuracy
trade-off specified by a hyperparameter. We evaluate the method on the MNIST,
CIFAR-10 and ImageNet classification benchmarks using six distinct model
architectures. Our results show that state-of-the-art model compression can be
achieved in a scalable and general way without requiring complex procedures
such as multi-stage training.
