Anchor & Transform: Learning Sparse Embeddings for Large Vocabularies
  Learning continuous representations of discrete objects such as text, users,
movies, and URLs lies at the heart of many applications including language and
user modeling. When using discrete objects as input to neural networks, we
often ignore the underlying structures (e.g., natural groupings and
similarities) and embed the objects independently into individual vectors. As a
result, existing methods do not scale to large vocabulary sizes. In this paper,
we design a simple and efficient embedding algorithm that learns a small set of
anchor embeddings and a sparse transformation matrix. We call our method Anchor
& Transform (ANT) as the embeddings of discrete objects are a sparse linear
combination of the anchors, weighted according to the transformation matrix.
ANT is scalable, flexible, and end-to-end trainable. We further provide a
statistical interpretation of our algorithm as a Bayesian nonparametric prior
for embeddings that encourages sparsity and leverages natural groupings among
objects. By deriving an approximate inference algorithm based on Small Variance
Asymptotics, we obtain a natural extension that automatically learns the
optimal number of anchors instead of having to tune it as a hyperparameter. On
text classification, language modeling, and movie recommendation benchmarks, we
show that ANT is particularly suitable for large vocabulary sizes and
demonstrates stronger performance with fewer parameters (up to 40x compression)
as compared to existing compression baselines.
