Riemannian adaptive stochastic gradient algorithms on matrix manifolds
  Adaptive stochastic gradient algorithms in the Euclidean space have attracted
much attention lately. Such explorations on Riemannian manifolds, on the other
hand, are relatively new, limited, and challenging. This is because of the
intrinsic non-linear structure of the underlying manifold and the absence of a
canonical coordinate system. In machine learning applications, however, most
manifolds of interest are represented as matrices with notions of row and
column subspaces. In addition, the implicit manifold-related constraints may
also lie on such subspaces. For example, the Grassmann manifold is the set of
column subspaces. To this end, such a rich structure should not be lost by
transforming matrices to just a stack of vectors while developing optimization
algorithms on manifolds. We propose novel stochastic gradient algorithms for
problems on Riemannian matrix manifolds by adapting the row and column
subspaces of gradients. Our algorithms are provably convergent and they achieve
the convergence rate of order $\mathcal{O}(\log (T)/\sqrt{T})$, where $T$ is
the number of iterations. Our experiments illustrate the efficacy of the
proposed algorithms on several applications.
