Bayesian Compression for Natural Language Processing
  In natural language processing, a lot of the tasks are successfully solved
with recurrent neural networks, but such models have a huge number of
parameters. The majority of these parameters are often concentrated in the
embedding layer, which size grows proportionally to the vocabulary length. We
propose a Bayesian sparsification technique for RNNs which allows compressing
the RNN dozens or hundreds of times without time-consuming hyperparameters
tuning. We also generalize the model for vocabulary sparsification to filter
out unnecessary words and compress the RNN even further. We show that the
choice of the kept words is interpretable. Code is available on github:
https://github.com/tipt0p/SparseBayesianRNN
