Randomized Block Frank-Wolfe for Convergent Large-Scale Learning
  Owing to their low-complexity iterations, Frank-Wolfe (FW) solvers are well
suited for various large-scale learning tasks. When block-separable constraints
are present, randomized block FW (RB-FW) has been shown to further reduce
complexity by updating only a fraction of coordinate blocks per iteration. To
circumvent the limitations of existing methods, the present work develops step
sizes for RB-FW that enable a flexible selection of the number of blocks to
update per iteration while ensuring convergence and feasibility of the
iterates. To this end, convergence rates of RB-FW are established through
computational bounds on a primal sub-optimality measure and on the duality gap.
The novel bounds extend the existing convergence analysis, which only applies
to a step-size sequence that does not generally lead to feasible iterates.
Furthermore, two classes of step-size sequences that guarantee feasibility of
the iterates are also proposed to enhance flexibility in choosing decay rates.
The novel convergence results are markedly broadened to encompass also
nonconvex objectives, and further assert that RB-FW with exact line-search
reaches a stationary point at rate $\mathcal{O}(1/\sqrt{t})$. Performance of
RB-FW with different step sizes and number of blocks is demonstrated in two
applications, namely charging of electrical vehicles and structural support
vector machines. Extensive simulated tests demonstrate the performance
improvement of RB-FW relative to existing randomized single-block FW methods.
