Minimal Random Code Learning: Getting Bits Back from Compressed Model
  Parameters
  While deep neural networks are a highly successful model class, their large
memory footprint puts considerable strain on energy consumption, communication
bandwidth, and storage requirements. Consequently, model size reduction has
become an utmost goal in deep learning. A typical approach is to train a set of
deterministic weights, while applying certain techniques such as pruning and
quantization, in order that the empirical weight distribution becomes amenable
to Shannon-style coding schemes. However, as shown in this paper, relaxing
weight determinism and using a full variational distribution over weights
allows for more efficient coding schemes and consequently higher compression
rates. In particular, following the classical bits-back argument, we encode the
network weights using a random sample, requiring only a number of bits
corresponding to the Kullback-Leibler divergence between the sampled
variational distribution and the encoding distribution. By imposing a
constraint on the Kullback-Leibler divergence, we are able to explicitly
control the compression rate, while optimizing the expected loss on the
training set. The employed encoding scheme can be shown to be close to the
optimal information-theoretical lower bound, with respect to the employed
variational family. Our method sets new state-of-the-art in neural network
compression, as it strictly dominates previous approaches in a Pareto sense: On
the benchmarks LeNet-5/MNIST and VGG-16/CIFAR-10, our approach yields the best
test performance for a fixed memory budget, and vice versa, it achieves the
highest compression rates for a fixed test performance.
