Adversarial Robustness via Fisher-Rao Regularization
  Adversarial robustness has become a topic of growing interest in machine
learning since it was observed that neural networks tend to be brittle. We
propose an information-geometric formulation of adversarial defense and
introduce FIRE, a new Fisher-Rao regularization for the categorical
cross-entropy loss, which is based on the geodesic distance between the softmax
outputs corresponding to natural and perturbed input features. Based on the
information-geometric properties of the class of softmax distributions, we
derive an explicit characterization of the Fisher-Rao Distance (FRD) for the
binary and multiclass cases, and draw some interesting properties as well as
connections with standard regularization metrics. Furthermore, for a simple
linear and Gaussian model, we show that all Pareto-optimal points in the
accuracy-robustness region can be reached by FIRE while other state-of-the-art
methods fail. Empirically, we evaluate the performance of various classifiers
trained with the proposed loss on standard datasets, showing up to a
simultaneous 1\% of improvement in terms of clean and robust performances while
reducing the training time by 20\% over the best-performing methods.
