Scalable Neural Network Compression and Pruning Using Hard Clustering
  and L1 Regularization
  We propose a simple and easy to implement neural network compression
algorithm that achieves results competitive with more complicated
state-of-the-art methods. The key idea is to modify the original optimization
problem by adding K independent Gaussian priors (corresponding to the k-means
objective) over the network parameters to achieve parameter quantization, as
well as an L1 penalty to achieve pruning. Unlike many existing
quantization-based methods, our method uses hard clustering assignments of
network parameters, which adds minimal change or overhead to standard network
training. We also demonstrate experimentally that tying neural network
parameters provides less gain in generalization performance than changing
network architecture and connectivity patterns entirely.
