Probabilistically Robust Learning: Balancing Average- and Worst-case
  Performance
  Many of the successes of machine learning are based on minimizing an averaged
loss function. However, it is well-known that this paradigm suffers from
robustness issues that hinder its applicability in safety-critical domains.
These issues are often addressed by training against worst-case perturbations
of data, a technique known as adversarial training. Although empirically
effective, adversarial training can be overly conservative, leading to
unfavorable trade-offs between nominal performance and robustness. To this end,
in this paper we propose a framework called probabilistic robustness that
bridges the gap between the accurate, yet brittle average case and the robust,
yet conservative worst case by enforcing robustness to most rather than to all
perturbations. From a theoretical point of view, this framework overcomes the
trade-offs between the performance and the sample-complexity of worst-case and
average-case learning. From a practical point of view, we propose a novel
algorithm based on risk-aware optimization that effectively balances average-
and worst-case performance at a considerably lower computational cost relative
to adversarial training. Our results on MNIST, CIFAR-10, and SVHN illustrate
the advantages of this framework on the spectrum from average- to worst-case
robustness.
