Convex Optimization: Algorithms and Complexity
  This monograph presents the main complexity theorems in convex optimization
and their corresponding algorithms. Starting from the fundamental theory of
black-box optimization, the material progresses towards recent advances in
structural optimization and stochastic optimization. Our presentation of
black-box optimization, strongly influenced by Nesterov's seminal book and
Nemirovski's lecture notes, includes the analysis of cutting plane methods, as
well as (accelerated) gradient descent schemes. We also pay special attention
to non-Euclidean settings (relevant algorithms include Frank-Wolfe, mirror
descent, and dual averaging) and discuss their relevance in machine learning.
We provide a gentle introduction to structural optimization with FISTA (to
optimize a sum of a smooth and a simple non-smooth term), saddle-point mirror
prox (Nemirovski's alternative to Nesterov's smoothing), and a concise
description of interior point methods. In stochastic optimization we discuss
stochastic gradient descent, mini-batches, random coordinate descent, and
sublinear algorithms. We also briefly touch upon convex relaxation of
combinatorial problems and the use of randomness to round solutions, as well as
random walks based methods.
