A Distributed Quasi-Newton Algorithm for Primal and Dual Regularized
  Empirical Risk Minimization
  We propose a communication- and computation-efficient distributed
optimization algorithm using second-order information for solving empirical
risk minimization (ERM) problems with a nonsmooth regularization term. Our
algorithm is applicable to both the primal and the dual ERM problem. Current
second-order and quasi-Newton methods for this problem either do not work well
in the distributed setting or work only for specific regularizers. Our
algorithm uses successive quadratic approximations of the smooth part, and we
describe how to maintain an approximation of the (generalized) Hessian and
solve subproblems efficiently in a distributed manner. When applied to the
distributed dual ERM problem, unlike state of the art that takes only the
block-diagonal part of the Hessian, our approach is able to utilize global
curvature information and is thus magnitudes faster. The proposed method enjoys
global linear convergence for a broad range of non-strongly convex problems
that includes the most commonly used ERMs, thus requiring lower communication
complexity. It also converges on non-convex problems, so has the potential to
be used on applications such as deep learning. Computational results
demonstrate that our method significantly improves on communication cost and
running time over the current state-of-the-art methods.
