Towards Sharp Stochastic Zeroth Order Hessian Estimators over Riemannian
  Manifolds
  We study Hessian estimators for real-valued functions defined over an
$n$-dimensional complete Riemannian manifold. We introduce new stochastic
zeroth-order Hessian estimators using $O (1)$ function evaluations. We show
that, for a smooth real-valued function $f$ with Lipschitz Hessian (with
respect to the Rimannian metric), our estimator achieves a bias bound of order
$ O \left( L_2 \delta + \gamma \delta^2 \right) $, where $ L_2 $ is the
Lipschitz constant for the Hessian, $ \gamma $ depends on both the Levi-Civita
connection and function $f$, and $\delta$ is the finite difference step size.
To the best of our knowledge, our results provide the first bias bound for
Hessian estimators that explicitly depends on the geometry of the underlying
Riemannian manifold. Perhaps more importantly, our bias bound does not increase
with dimension $n$. This improves best previously known bias bound for
$O(1)$-evaluation Hessian estimators, which increases quadratically with $n$.
We also study downstream computations based on our Hessian estimators. The
supremacy of our method is evidenced by empirical evaluations.
