SEBOOST - Boosting Stochastic Learning Using Subspace Optimization
  Techniques
  We present SEBOOST, a technique for boosting the performance of existing
stochastic optimization methods. SEBOOST applies a secondary optimization
process in the subspace spanned by the last steps and descent directions. The
method was inspired by the SESOP optimization method for large-scale problems,
and has been adapted for the stochastic learning framework. It can be applied
on top of any existing optimization method with no need to tweak the internal
algorithm. We show that the method is able to boost the performance of
different algorithms, and make them more robust to changes in their
hyper-parameters. As the boosting steps of SEBOOST are applied between large
sets of descent steps, the additional subspace optimization hardly increases
the overall computational burden. We introduce two hyper-parameters that
control the balance between the baseline method and the secondary optimization
process. The method was evaluated on several deep learning tasks, demonstrating
promising results.
