Robust Accelerated Gradient Methods for Smooth Strongly Convex Functions
  We study the trade-offs between convergence rate and robustness to gradient
errors in designing a first-order algorithm. We focus on gradient descent (GD)
and accelerated gradient (AG) methods for minimizing strongly convex functions
when the gradient has random errors in the form of additive white noise. With
gradient errors, the function values of the iterates need not converge to the
optimal value; hence, we define the robustness of an algorithm to noise as the
asymptotic expected suboptimality of the iterate sequence to input noise power.
For this robustness measure, we provide exact expressions for the quadratic
case using tools from robust control theory and tight upper bounds for the
smooth strongly convex case using Lyapunov functions certified through matrix
inequalities. We use these characterizations within an optimization problem
which selects parameters of each algorithm to achieve a particular trade-off
between rate and robustness. Our results show that AG can achieve acceleration
while being more robust to random gradient errors. This behavior is quite
different than previously reported in the deterministic gradient noise setting.
We also establish some connections between the robustness of an algorithm and
how quickly it can converge back to the optimal solution if it is perturbed
from the optimal point with deterministic noise. Our framework also leads to
practical algorithms that can perform better than other state-of-the-art
methods in the presence of random gradient noise.
