Model compression as constrained optimization, with application to
  neural nets. Part I: general framework
  Compressing neural nets is an active research problem, given the large size
of state-of-the-art nets for tasks such as object recognition, and the
computational limits imposed by mobile devices. We give a general formulation
of model compression as constrained optimization. This includes many types of
compression: quantization, low-rank decomposition, pruning, lossless
compression and others. Then, we give a general algorithm to optimize this
nonconvex problem based on the augmented Lagrangian and alternating
optimization. This results in a "learning-compression" algorithm, which
alternates a learning step of the uncompressed model, independent of the
compression type, with a compression step of the model parameters, independent
of the learning task. This simple, efficient algorithm is guaranteed to find
the best compressed model for the task in a local sense under standard
assumptions.
  We present separately in several companion papers the development of this
general framework into specific algorithms for model compression based on
quantization, pruning and other variations, including experimental results on
compressing neural nets and other models.
