On the regularized risk of distributionally robust learning over deep
  neural networks
  In this paper we explore the relation between distributionally robust
learning and different forms of regularization to enforce robustness of deep
neural networks. In particular, starting from a concrete min-max
distributionally robust problem, and using tools from optimal transport theory,
we derive first order and second order approximations to the distributionally
robust problem in terms of appropriate regularized risk minimization problems.
In the context of deep ResNet models, we identify the structure of the
resulting regularization problems as mean-field optimal control problems where
the number and dimension of state variables is within a dimension-free factor
of the dimension of the original unrobust problem. Using the Pontryagin maximum
principles associated to these problems we motivate a family of scalable
algorithms for the training of robust neural networks. Our analysis recovers
some results and algorithms known in the literature (in settings explained
throughout the paper) and provides many other theoretical and algorithmic
insights that to our knowledge are novel. In our analysis we employ tools that
we deem useful for a future analysis of more general adversarial learning
problems.
