SlimNets: An Exploration of Deep Model Compression and Acceleration
  Deep neural networks have achieved increasingly accurate results on a wide
variety of complex tasks. However, much of this improvement is due to the
growing use and availability of computational resources (e.g use of GPUs, more
layers, more parameters, etc). Most state-of-the-art deep networks, despite
performing well, over-parameterize approximate functions and take a significant
amount of time to train. With increased focus on deploying deep neural networks
on resource constrained devices like smart phones, there has been a push to
evaluate why these models are so resource hungry and how they can be made more
efficient. This work evaluates and compares three distinct methods for deep
model compression and acceleration: weight pruning, low rank factorization, and
knowledge distillation. Comparisons on VGG nets trained on CIFAR10 show that
each of the models on their own are effective, but that the true power lies in
combining them. We show that by combining pruning and knowledge distillation
methods we can create a compressed network 85 times smaller than the original,
all while retaining 96% of the original model's accuracy.
