Training ReLU networks to high uniform accuracy is intractable
  Statistical learning theory provides bounds on the necessary number of
training samples needed to reach a prescribed accuracy in a learning problem
formulated over a given target class. This accuracy is typically measured in
terms of a generalization error, that is, an expected value of a given loss
function. However, for several applications -- for example in a
security-critical context or for problems in the computational sciences --
accuracy in this sense is not sufficient. In such cases, one would like to have
guarantees for high accuracy on every input value, that is, with respect to the
uniform norm. In this paper we precisely quantify the number of training
samples needed for any conceivable training algorithm to guarantee a given
uniform accuracy on any learning problem formulated over target classes
containing (or consisting of) ReLU neural networks of a prescribed
architecture. We prove that, under very general assumptions, the minimal number
of training samples for this task scales exponentially both in the depth and
the input dimension of the network architecture. As a corollary we conclude
that the training of ReLU neural networks to high uniform accuracy is
intractable. In a security-critical context this points to the fact that deep
learning based systems are prone to being fooled by a possible adversary. We
corroborate our theoretical findings by numerical results.
