LilNetX: Lightweight Networks with EXtreme Model Compression and
  Structured Sparsification
  We introduce LilNetX, an end-to-end trainable technique for neural networks
that enables learning models with specified accuracy-rate-computation
trade-off. Prior works approach these problems one at a time and often require
post-processing or multistage training which become less practical and do not
scale very well for large datasets or architectures. Our method constructs a
joint training objective that penalizes the self-information of network
parameters in a reparameterized latent space to encourage small model size
while also introducing priors to increase structured sparsity in the parameter
space to reduce computation. We achieve up to 50% smaller model size and 98%
model sparsity on ResNet-20 while retaining the same accuracy on the CIFAR-10
dataset as well as 35% smaller model size and 42% structured sparsity on
ResNet-50 trained on ImageNet, when compared to existing state-of-the-art model
compression methods. Code is available at
https://github.com/Sharath-girish/LilNetX.
