$k$FW: A Frank-Wolfe style algorithm with stronger subproblem oracles
  This paper proposes a new variant of Frank-Wolfe (FW), called $k$FW. Standard
FW suffers from slow convergence: iterates often zig-zag as update directions
oscillate around extreme points of the constraint set. The new variant, $k$FW,
overcomes this problem by using two stronger subproblem oracles in each
iteration. The first is a $k$ linear optimization oracle ($k$LOO) that computes
the $k$ best update directions (rather than just one). The second is a $k$
direction search ($k$DS) that minimizes the objective over a constraint set
represented by the $k$ best update directions and the previous iterate. When
the problem solution admits a sparse representation, both oracles are easy to
compute, and $k$FW converges quickly for smooth convex objectives and several
interesting constraint sets: $k$FW achieves finite
$\frac{4L_f^3D^4}{\gamma\delta^2}$ convergence on polytopes and group norm
balls, and linear convergence on spectrahedra and nuclear norm balls. Numerical
experiments validate the effectiveness of $k$FW and demonstrate an
order-of-magnitude speedup over existing approaches.
