Scaling Up Sparse Support Vector Machines by Simultaneous Feature and
  Sample Reduction
  Sparse support vector machine (SVM) is a popular classification technique
that can simultaneously learn a small set of the most interpretable features
and identify the support vectors. It has achieved great successes in many
real-world applications. However, for large-scale problems involving a huge
number of samples and ultra-high dimensional features, solving sparse SVMs
remains challenging. By noting that sparse SVMs induce sparsities in both
feature and sample spaces, we propose a novel approach, which is based on
accurate estimations of the primal and dual optima of sparse SVMs, to
simultaneously identify the inactive features and samples that are guaranteed
to be irrelevant to the outputs. Thus, we can remove the identified inactive
samples and features from the training phase, leading to substantial savings in
the computational cost without sacrificing the accuracy. Moreover, we show that
our method can be extended to multi-class sparse support vector machines. To
the best of our knowledge, the proposed method is the \emph{first}
\emph{static} feature and sample reduction method for sparse SVMs and
multi-class sparse SVMs. Experiments on both synthetic and real data sets
demonstrate that our approach significantly outperforms state-of-the-art
methods and the speedup gained by our approach can be orders of magnitude.
