Information-Theoretic Understanding of Population Risk Improvement with
  Model Compression
  We show that model compression can improve the population risk of a
pre-trained model, by studying the tradeoff between the decrease in the
generalization error and the increase in the empirical risk with model
compression. We first prove that model compression reduces an
information-theoretic bound on the generalization error; this allows for an
interpretation of model compression as a regularization technique to avoid
overfitting. We then characterize the increase in empirical risk with model
compression using rate distortion theory. These results imply that the
population risk could be improved by model compression if the decrease in
generalization error exceeds the increase in empirical risk. We show through a
linear regression example that such a decrease in population risk due to model
compression is indeed possible. Our theoretical results further suggest that
the Hessian-weighted $K$-means clustering compression approach can be improved
by regularizing the distance between the clustering centers. We provide
experiments with neural networks to support our theoretical assertions.
