Gaussian Robust Classification
  Supervised learning is all about the ability to generalize knowledge.
Specifically, the goal of the learning is to train a classifier using training
data, in such a way that it will be capable of classifying new unseen data
correctly. In order to acheive this goal, it is important to carefully design
the learner, so it will not overfit the training data. The later can is done
usually by adding a regularization term. The statistical learning theory
explains the success of this method by claiming that it restricts the
complexity of the learned model. This explanation, however, is rather abstract
and does not have a geometric intuition. The generalization error of a
classifier may be thought of as correlated with its robustness to perturbations
of the data: a classifier that copes with disturbance is expected to generalize
well. Indeed, Xu et al. [2009] have shown that the SVM formulation is
equivalent to a robust optimization (RO) formulation, in which an adversary
displaces the training and testing points within a ball of pre-determined
radius. In this work we explore a different kind of robustness, namely changing
each data point with a Gaussian cloud centered at the sample. Loss is evaluated
as the expectation of an underlying loss function on the cloud. This setup fits
the fact that in many applications, the data is sampled along with noise. We
develop an RO framework, in which the adversary chooses the covariance of the
noise. In our algorithm named GURU, the tuning parameter is a spectral bound on
the noise, thus it can be estimated using physical or applicative
considerations. Our experiments show that this framework performs as well as
SVM and even slightly better in some cases. Generalizations for Mercer kernels
and for the multiclass case are presented as well. We also show that our
framework may be further generalized, using the technique of convex perspective
functions.
