On the Global Linear Convergence of Frank-Wolfe Optimization Variants
  The Frank-Wolfe (FW) optimization algorithm has lately re-gained popularity
thanks in particular to its ability to nicely handle the structured constraints
appearing in machine learning applications. However, its convergence rate is
known to be slow (sublinear) when the solution lies at the boundary. A simple
less-known fix is to add the possibility to take 'away steps' during
optimization, an operation that importantly does not require a feasibility
oracle. In this paper, we highlight and clarify several variants of the
Frank-Wolfe optimization algorithm that have been successfully applied in
practice: away-steps FW, pairwise FW, fully-corrective FW and Wolfe's minimum
norm point algorithm, and prove for the first time that they all enjoy global
linear convergence, under a weaker condition than strong convexity of the
objective. The constant in the convergence rate has an elegant interpretation
as the product of the (classical) condition number of the function with a novel
geometric quantity that plays the role of a 'condition number' of the
constraint set. We provide pointers to where these algorithms have made a
difference in practice, in particular with the flow polytope, the marginal
polytope and the base polytope for submodular optimization.
