PASSCoDe: Parallel ASynchronous Stochastic dual Co-ordinate Descent
  Stochastic Dual Coordinate Descent (SDCD) has become one of the most
efficient ways to solve the family of $\ell_2$-regularized empirical risk
minimization problems, including linear SVM, logistic regression, and many
others. The vanilla implementation of DCD is quite slow; however, by
maintaining primal variables while updating dual variables, the time complexity
of SDCD can be significantly reduced. Such a strategy forms the core algorithm
in the widely-used LIBLINEAR package. In this paper, we parallelize the SDCD
algorithms in LIBLINEAR. In recent research, several synchronized parallel SDCD
algorithms have been proposed, however, they fail to achieve good speedup in
the shared memory multi-core setting. In this paper, we propose a family of
asynchronous stochastic dual coordinate descent algorithms (ASDCD). Each thread
repeatedly selects a random dual variable and conducts coordinate updates using
the primal variables that are stored in the shared memory. We analyze the
convergence properties when different locking/atomic mechanisms are applied.
For implementation with atomic operations, we show linear convergence under
mild conditions. For implementation without any atomic operations or locking,
we present the first {\it backward error analysis} for ASDCD under the
multi-core environment, showing that the converged solution is the exact
solution for a primal problem with perturbed regularizer. Experimental results
show that our methods are much faster than previous parallel coordinate descent
solvers.
