A Robust Optimization Approach to Deep Learning
  Many state-of-the-art adversarial training methods leverage upper bounds of
the adversarial loss to provide security guarantees. Yet, these methods require
computations at each training step that can not be incorporated in the gradient
for backpropagation. We introduce a new, more principled approach to
adversarial training based on a closed form solution of an upper bound of the
adversarial loss, which can be effectively trained with backpropagation. This
bound is facilitated by state-of-the-art tools from robust optimization. We
derive two new methods with our approach. The first method (Approximated Robust
Upper Bound or aRUB) uses the first order approximation of the network as well
as basic tools from linear robust optimization to obtain an approximate upper
bound of the adversarial loss that can be easily implemented. The second method
(Robust Upper Bound or RUB), computes an exact upper bound of the adversarial
loss. Across a variety of tabular and vision data sets we demonstrate the
effectiveness of our more principled approach -- RUB is substantially more
robust than state-of-the-art methods for larger perturbations, while aRUB
matches the performance of state-of-the-art methods for small perturbations.
Also, both RUB and aRUB run faster than standard adversarial training (at the
expense of an increase in memory). All the code to reproduce the results can be
found at https://github.com/kimvc7/Robustness.
