Democratic Source Coding: An Optimal Fixed-Length Quantization Scheme
  for Distributed Optimization Under Communication Constraints
  The communication cost of distributed optimization algorithms is a major
bottleneck in their scalability. This work considers a parameter-server setting
in which the worker is constrained to communicate information to the server
using only $R$ bits per dimension. We show that $\mathbf{democratic}$
$\mathbf{embeddings}$ from random matrix theory are significantly useful for
designing efficient and optimal vector quantizers that respect this bit budget.
The resulting polynomial complexity source coding schemes are used to design
distributed optimization algorithms with convergence rates matching the minimax
optimal lower bounds for (i) Smooth and Strongly-Convex objectives with access
to an Exact Gradient oracle, as well as (ii) General Convex and Non-Smooth
objectives with access to a Noisy Subgradient oracle. We further propose a
relaxation of this coding scheme which is nearly minimax optimal. Numerical
simulations validate our theoretical claims.
