Preconditioned Riemannian Optimization on the Generalized Stiefel
  Manifold
  Optimization problems on the generalized Stiefel manifold (and products of
it) are prevalent across science and engineering. For example, in computational
science they arise in the symmetric (generalized) eigenvalue problem, in
nonlinear eigenvalue problems, and in electronic structures computations, to
name a few problems. In statistics and machine learning, they arise, for
example, in various dimensionality reduction techniques such as canonical
correlation analysis. In deep learning, regularization and improved stability
can be obtained by constraining some layers to have parameter matrices that
belong to the Stiefel manifold. Solving problems on the generalized Stiefel
manifold can be approached via the tools of Riemannian optimization. However,
using the standard geometric components for the generalized Stiefel manifold
has two possible shortcoming: computing some of the geometric components can be
too expensive and converge can be rather slow in certain cases. Both
shortcomings can be addressed using a technique called Riemannian
preconditioning, which amounts to using geometric components derived using a
precoditioner that defines a Riemannian metric on the constraint manifold. In
this paper we develop the geometric components required to perform Riemannian
optimization on the generalized Stiefel manifold equipped with a non-standard
metric, and illustrate theoretically and numerically the use of those
components and the effect of Riemannian preconditioning for solving
optimization problems on the generalized Stiefel manifold.
