Improving Optimization for Models With Continuous Symmetry Breaking
  Many loss functions in representation learning are invariant under a
continuous symmetry transformation. For example, the loss function of word
embeddings (Mikolov et al., 2013) remains unchanged if we simultaneously rotate
all word and context embedding vectors. We show that representation learning
models for time series possess an approximate continuous symmetry that leads to
slow convergence of gradient descent. We propose a new optimization algorithm
that speeds up convergence using ideas from gauge theory in physics. Our
algorithm leads to orders of magnitude faster convergence and to more
interpretable representations, as we show for dynamic extensions of matrix
factorization and word embedding models. We further present an example
application of our proposed algorithm that translates modern words into their
historic equivalents.
