Communication-Efficient Distributed Stochastic AUC Maximization with
  Deep Neural Networks
  In this paper, we study distributed algorithms for large-scale AUC
maximization with a deep neural network as a predictive model. Although
distributed learning techniques have been investigated extensively in deep
learning, they are not directly applicable to stochastic AUC maximization with
deep neural networks due to its striking differences from standard loss
minimization problems (e.g., cross-entropy). Towards addressing this challenge,
we propose and analyze a communication-efficient distributed optimization
algorithm based on a {\it non-convex concave} reformulation of the AUC
maximization, in which the communication of both the primal variable and the
dual variable between each worker and the parameter server only occurs after
multiple steps of gradient-based updates in each worker. Compared with the
naive parallel version of an existing algorithm that computes stochastic
gradients at individual machines and averages them for updating the model
parameters, our algorithm requires a much less number of communication rounds
and still achieves a linear speedup in theory. To the best of our knowledge,
this is the \textbf{first} work that solves the {\it non-convex concave
min-max} problem for AUC maximization with deep neural networks in a
communication-efficient distributed manner while still maintaining the linear
speedup property in theory. Our experiments on several benchmark datasets show
the effectiveness of our algorithm and also confirm our theory.
