On Convergence of Distributed Approximate Newton Methods: Globalization,
  Sharper Bounds and Beyond
  The DANE algorithm is an approximate Newton method popularly used for
communication-efficient distributed machine learning. Reasons for the interest
in DANE include scalability and versatility. Convergence of DANE, however, can
be tricky; its appealing convergence rate is only rigorous for quadratic
objective, and for more general convex functions the known results are no
stronger than those of the classic first-order methods. To remedy these
drawbacks, we propose in this paper some new alternatives of DANE which are
more suitable for analysis. We first introduce a simple variant of DANE
equipped with backtracking line search, for which global asymptotic convergence
and sharper local non-asymptotic convergence rate guarantees can be proved for
both quadratic and non-quadratic strongly convex functions. Then we propose a
heavy-ball method to accelerate the convergence of DANE, showing that nearly
tight local rate of convergence can be established for strongly convex
functions, and with proper modification of algorithm the same result applies
globally to linear prediction models. Numerical evidence is provided to confirm
the theoretical and practical advantages of our methods.
