Invariance and identifiability issues for word embeddings
  Word embeddings are commonly obtained as optimizers of a criterion function
$f$ of a text corpus, but assessed on word-task performance using a different
evaluation function $g$ of the test data. We contend that a possible source of
disparity in performance on tasks is the incompatibility between classes of
transformations that leave $f$ and $g$ invariant. In particular, word
embeddings defined by $f$ are not unique; they are defined only up to a class
of transformations to which $f$ is invariant, and this class is larger than the
class to which $g$ is invariant. One implication of this is that the apparent
superiority of one word embedding over another, as measured by word task
performance, may largely be a consequence of the arbitrary elements selected
from the respective solution sets. We provide a formal treatment of the above
identifiability issue, present some numerical examples, and discuss possible
resolutions.
