Trivializations for Gradient-Based Optimization on Manifolds
  We introduce a framework to study the transformation of problems with
manifold constraints into unconstrained problems through parametrizations in
terms of a Euclidean space. We call these parametrizations "trivializations".
We prove conditions under which a trivialization is sound in the context of
gradient-based optimization and we show how two large families of
trivializations have overall favorable properties, but also suffer from a
performance issue. We then introduce "dynamic trivializations", which solve
this problem, and we show how these form a family of optimization methods that
lie between trivializations and Riemannian gradient descent, and combine the
benefits of both of them. We then show how to implement these two families of
trivializations in practice for different matrix manifolds. To this end, we
prove a formula for the gradient of the exponential of matrices, which can be
of practical interest on its own. Finally, we show how dynamic trivializations
improve the performance of existing methods on standard tasks designed to test
long-term memory within neural networks.
