SDNA: Stochastic Dual Newton Ascent for Empirical Risk Minimization
  We propose a new algorithm for minimizing regularized empirical loss:
Stochastic Dual Newton Ascent (SDNA). Our method is dual in nature: in each
iteration we update a random subset of the dual variables. However, unlike
existing methods such as stochastic dual coordinate ascent, SDNA is capable of
utilizing all curvature information contained in the examples, which leads to
striking improvements in both theory and practice - sometimes by orders of
magnitude. In the special case when an L2-regularizer is used in the primal,
the dual problem is a concave quadratic maximization problem plus a separable
term. In this regime, SDNA in each step solves a proximal subproblem involving
a random principal submatrix of the Hessian of the quadratic function; whence
the name of the method. If, in addition, the loss functions are quadratic, our
method can be interpreted as a novel variant of the recently introduced
Iterative Hessian Sketch.
