On the Compression of Natural Language Models
  Deep neural networks are effective feature extractors but they are
prohibitively large for deployment scenarios. Due to the huge number of
parameters, interpretability of parameters in different layers is not
straight-forward. This is why neural networks are sometimes considered black
boxes. Although simpler models are easier to explain, finding them is not easy.
If found, a sparse network that can fit to a data from scratch would help to
interpret parameters of a neural network. To this end, lottery ticket
hypothesis states that typical dense neural networks contain a small sparse
sub-network that can be trained to a reach similar test accuracy in an equal
number of steps. The goal of this work is to assess whether such a trainable
subnetwork exists for natural language models (NLM)s. To achieve this goal we
will review state-of-the-art compression techniques such as quantization,
knowledge distillation, and pruning.
