Communication-Efficient Distributed Optimization of Self-Concordant
  Empirical Loss
  We consider distributed convex optimization problems originated from sample
average approximation of stochastic optimization, or empirical risk
minimization in machine learning. We assume that each machine in the
distributed computing system has access to a local empirical loss function,
constructed with i.i.d. data sampled from a common distribution. We propose a
communication-efficient distributed algorithm to minimize the overall empirical
loss, which is the average of the local empirical losses. The algorithm is
based on an inexact damped Newton method, where the inexact Newton steps are
computed by a distributed preconditioned conjugate gradient method. We analyze
its iteration complexity and communication efficiency for minimizing
self-concordant empirical loss functions, and discuss the results for
distributed ridge regression, logistic regression and binary classification
with a smoothed hinge loss. In a standard setting for supervised learning, the
required number of communication rounds of the algorithm does not increase with
the sample size, and only grows slowly with the number of machines.
