Analysis of Optimization Algorithms via Sum-of-Squares
  We introduce a new framework for unifying and systematizing the performance
analysis of first-order black-box optimization algorithms for unconstrained
convex minimization. The low-cost iteration complexity enjoyed by first-order
algorithms renders them particularly relevant for applications in machine
learning and large-scale data analysis. Relying on sum-of-squares (SOS)
optimization, we introduce a hierarchy of semidefinite programs that give
increasingly better convergence bounds for higher levels of the hierarchy.
Alluding to the power of the SOS hierarchy, we show that the (dual of the)
first level corresponds to the Performance Estimation Problem (PEP) introduced
by Drori and Teboulle [Math. Program., 145(1):451--482, 2014], a powerful
framework for determining convergence rates of first-order optimization
algorithms. Consequently, many results obtained within the PEP framework can be
reinterpreted as degree-1 SOS proofs, and thus, the SOS framework provides a
promising new approach for certifying improved rates of convergence by means of
higher-order SOS certificates. To determine analytical rate bounds, in this
work we use the first level of the SOS hierarchy and derive new result{s} for
noisy gradient descent with inexact line search methods (Armijo, Wolfe, and
Goldstein).
