Blended Conditional Gradients: the unconditioning of conditional
  gradients
  We present a blended conditional gradient approach for minimizing a smooth
convex function over a polytope P, combining the Frank--Wolfe algorithm (also
called conditional gradient) with gradient-based steps, different from away
steps and pairwise steps, but still achieving linear convergence for strongly
convex functions, along with good practical performance. Our approach retains
all favorable properties of conditional gradient algorithms, notably avoidance
of projections onto P and maintenance of iterates as sparse convex combinations
of a limited number of extreme points of P. The algorithm is lazy, making use
of inexpensive inexact solutions of the linear programming subproblem that
characterizes the conditional gradient approach. It decreases measures of
optimality (primal and dual gaps) rapidly, both in the number of iterations and
in wall-clock time, outperforming even the lazy conditional gradient algorithms
of [arXiv:1410.8816]. We also present a streamlined version of the algorithm
for the probability simplex.
