Learning Games and Rademacher Observations Losses
  It has recently been shown that supervised learning with the popular logistic
loss is equivalent to optimizing the exponential loss over sufficient
statistics about the class: Rademacher observations (rados). We first show that
this unexpected equivalence can actually be generalized to other example / rado
losses, with necessary and sufficient conditions for the equivalence,
exemplified on four losses that bear popular names in various fields:
exponential (boosting), mean-variance (finance), Linear Hinge (on-line
learning), ReLU (deep learning), and unhinged (statistics). Second, we show
that the generalization unveils a surprising new connection to regularized
learning, and in particular a sufficient condition under which regularizing the
loss over examples is equivalent to regularizing the rados (with Minkowski
sums) in the equivalent rado loss. This brings simple and powerful rado-based
learning algorithms for sparsity-controlling regularization, that we exemplify
on a boosting algorithm for the regularized exponential rado-loss, which
formally boosts over four types of regularization, including the popular ridge
and lasso, and the recently coined slope --- we obtain the first proven
boosting algorithm for this last regularization. Through our first contribution
on the equivalence of rado and example-based losses, Omega-R.AdaBoost~appears
to be an efficient proxy to boost the regularized logistic loss over examples
using whichever of the four regularizers. Experiments display that
regularization consistently improves performances of rado-based learning, and
may challenge or beat the state of the art of example-based learning even when
learning over small sets of rados. Finally, we connect regularization to
differential privacy, and display how tiny budgets can be afforded on big
domains while beating (protected) example-based learning.
