The Minimax Complexity of Distributed Optimization
  In this thesis, I study the minimax oracle complexity of distributed
stochastic optimization. First, I present the "graph oracle model", an
extension of the classic oracle complexity framework that can be applied to
study distributed optimization algorithms. Next, I describe a general approach
to proving optimization lower bounds for arbitrary randomized algorithms (as
opposed to more restricted classes of algorithms, e.g., deterministic or
"zero-respecting" algorithms), which is used extensively throughout the thesis.
For the remainder of the thesis, I focus on the specific case of the
"intermittent communication setting", where multiple computing devices work in
parallel with limited communication amongst themselves. In this setting, I
analyze the theoretical properties of the popular Local Stochastic Gradient
Descent (SGD) algorithm in convex setting, both for homogeneous and
heterogeneous objectives. I provide the first guarantees for Local SGD that
improve over simple baseline methods, but show that Local SGD is not optimal in
general. In pursuit of optimal methods in the intermittent communication
setting, I then show matching upper and lower bounds for the intermittent
communication setting with homogeneous convex, heterogeneous convex, and
homogeneous non-convex objectives. These upper bounds are attained by simple
variants of SGD which are therefore optimal. Finally, I discuss several
additional assumptions about the objective or more powerful oracles that might
be exploitable in order to develop better intermittent communication algorithms
with better guarantees than our lower bounds allow.
