Structured Sparsification of Gated Recurrent Neural Networks
  Recently, a lot of techniques were developed to sparsify the weights of
neural networks and to remove networks' structure units, e.g. neurons. We
adjust the existing sparsification approaches to the gated recurrent
architectures. Specifically, in addition to the sparsification of weights and
neurons, we propose sparsifying the preactivations of gates. This makes some
gates constant and simplifies LSTM structure. We test our approach on the text
classification and language modeling tasks. We observe that the resulting
structure of gate sparsity depends on the task and connect the learned
structure to the specifics of the particular tasks. Our method also improves
neuron-wise compression of the model in most of the tasks.
