The Stochastic Gradient Descent for the Primal L1-SVM Optimization
  Revisited
  We reconsider the stochastic (sub)gradient approach to the unconstrained
primal L1-SVM optimization. We observe that if the learning rate is inversely
proportional to the number of steps, i.e., the number of times any training
pattern is presented to the algorithm, the update rule may be transformed into
the one of the classical perceptron with margin in which the margin threshold
increases linearly with the number of steps. Moreover, if we cycle repeatedly
through the possibly randomly permuted training set the dual variables defined
naturally via the expansion of the weight vector as a linear combination of the
patterns on which margin errors were made are shown to obey at the end of each
complete cycle automatically the box constraints arising in dual optimization.
This renders the dual Lagrangian a running lower bound on the primal objective
tending to it at the optimum and makes available an upper bound on the relative
accuracy achieved which provides a meaningful stopping criterion. In addition,
we propose a mechanism of presenting the same pattern repeatedly to the
algorithm which maintains the above properties. Finally, we give experimental
evidence that algorithms constructed along these lines exhibit a considerably
improved performance.
