Distributed Sketching for Randomized Optimization: Exact
  Characterization, Concentration and Lower Bounds
  We consider distributed optimization methods for problems where forming the
Hessian is computationally challenging and communication is a significant
bottleneck. We leverage randomized sketches for reducing the problem dimensions
as well as preserving privacy and improving straggler resilience in
asynchronous distributed systems. We derive novel approximation guarantees for
classical sketching methods and establish tight concentration results that
serve as both upper and lower bounds on the error. We then extend our analysis
to the accuracy of parameter averaging for distributed sketches. Furthermore,
we develop unbiased parameter averaging methods for randomized second order
optimization for regularized problems that employ sketching of the Hessian.
Existing works do not take the bias of the estimators into consideration, which
limits their application to massively parallel computation. We provide
closed-form formulas for regularization parameters and step sizes that provably
minimize the bias for sketched Newton directions. Additionally, we demonstrate
the implications of our theoretical findings via large scale experiments on a
serverless cloud computing platform.
