On Geometric Connections of Embedded and Quotient Geometries in
  Riemannian Fixed-rank Matrix Optimization
  In this paper, we propose a general procedure for establishing the landscape
connections of a Riemannian optimization problem under the embedded and
quotient geometries. By applying the general procedure to the fixed-rank
positive semidefinite (PSD) and general matrix optimization, we establish an
exact Riemannian gradient connection under two geometries at every point on the
manifold and sandwich inequalities between the spectra of Riemannian Hessians
at Riemannian first-order stationary points (FOSPs). These results immediately
imply an equivalence on the sets of Riemannian FOSPs, Riemannian second-order
stationary points (SOSPs) and strict saddles of fixed-rank matrix optimization
under the embedded and the quotient geometries. To the best of our knowledge,
this is the first geometric landscape connection between the embedded and the
quotient geometries for fixed-rank matrix optimization and it provides a
concrete example on how these two geometries are connected in Riemannian
optimization. In addition, the effects of the Riemannian metric and quotient
structure on the landscape connection are discussed. We also observe an
algorithmic connection for fixed-rank matrix optimization under two geometries
with some specific Riemannian metrics. A number of novel ideas and technical
ingredients including a unified treatment for different Riemannian metrics and
new horizontal space representations under quotient geometries are developed to
obtain our results. The results in this paper deepen our understanding of
geometric connections of Riemannian optimization under different Riemannian
geometries and provide a few new theoretical insights to unanswered questions
in the literature.
