Momentum Stiefel Optimizer, with Applications to Suitably-Orthogonal
  Attention, and Optimal Transport
  The problem of optimization on Stiefel manifold, i.e., minimizing functions
of (not necessarily square) matrices that satisfy orthogonality constraints,
has been extensively studied, partly due to rich machine learning applications.
Yet, a new approach is proposed based on, for the first time, an interplay
between thoughtfully designed continuous and discrete dynamics. It leads to a
gradient-based optimizer with intrinsically added momentum. This method exactly
preserves the manifold structure but does not require commonly used projection
or retraction, and thus having low computational costs when compared to
existing algorithms. Its generalization to adaptive learning rates is also
demonstrated. Pleasant performances are observed in various practical tasks.
For instance, we discover that placing orthogonal constraints on attention
heads of trained-from-scratch Vision Transformer [Dosovitskiy et al. 2022]
could remarkably improve its performance, when our optimizer is used, and it is
better that each head is made orthogonal within itself but not necessarily to
other heads. This optimizer also makes the useful notion of Projection Robust
Wasserstein Distance [Paty & Cuturi 2019][Lin et al. 2020] for high-dim.
optimal transport even more effective.
