Deconstructing and reconstructing word embedding algorithms
  Uncontextualized word embeddings are reliable feature representations of
words used to obtain high quality results for various NLP applications. Given
the historical success of word embeddings in NLP, we propose a retrospective on
some of the most well-known word embedding algorithms. In this work, we
deconstruct Word2vec, GloVe, and others, into a common form, unveiling some of
the necessary and sufficient conditions required for making performant word
embeddings. We find that each algorithm: (1) fits vector-covector dot products
to approximate pointwise mutual information (PMI); and, (2) modulates the loss
gradient to balance weak and strong signals. We demonstrate that these two
algorithmic features are sufficient conditions to construct a novel word
embedding algorithm, Hilbert-MLE. We find that its embeddings obtain equivalent
or better performance against other algorithms across 17 intrinsic and
extrinsic datasets.
