Compressibility of Distributed Document Representations
  Contemporary natural language processing (NLP) revolves around learning from
latent document representations, generated either implicitly by neural language
models or explicitly by methods such as doc2vec or similar. One of the key
properties of the obtained representations is their dimension. Whilst the
commonly adopted dimensions of 256 and 768 offer sufficient performance on many
tasks, it is many times unclear whether the default dimension is the most
suitable choice for the subsequent downstream learning tasks. Furthermore,
representation dimensions are seldom subject to hyperparameter tuning due to
computational constraints. The purpose of this paper is to demonstrate that a
surprisingly simple and efficient recursive compression procedure can be
sufficient to both significantly compress the initial representation, but also
potentially improve its performance when considering the task of text
classification. Having smaller and less noisy representations is the desired
property during deployment, as orders of magnitude smaller models can
significantly reduce the computational overload and with it the deployment
costs. We propose CoRe, a straightforward, representation learner-agnostic
framework suitable for representation compression. The CoRe's performance is
showcased and studied on a collection of 17 real-life corpora from biomedical,
news, social media, and literary domains. We explored CoRe's behavior when
considering contextual and non-contextual document representations, different
compression levels, and 9 different compression algorithms. Current results
based on more than 100,000 compression experiments indicate that recursive
Singular Value Decomposition offers a very good trade-off between the
compression efficiency and performance, making CoRe useful in many existing,
representation-dependent NLP pipelines.
