The Probabilistic Fault Tolerance of Neural Networks in the Continuous
  Limit
  The loss of a few neurons in a brain rarely results in any visible loss of
function. However, the insight into what "few" means in this context is
unclear. How many random neuron failures will it take to lead to a visible loss
of function? In this paper, we address the fundamental question of the impact
of the crash of a random subset of neurons on the overall computation of a
neural network and the error in the output it produces. We study fault
tolerance of neural networks subject to small random neuron/weight crash
failures in a probabilistic setting. We give provable guarantees on the
robustness of the network to these crashes. Our main contribution is a bound on
the error in the output of a network under small random Bernoulli crashes
proved by using a Taylor expansion in the continuous limit, where close-by
neurons at a layer are similar. The failure mode we adopt in our model is
characteristic of neuromorphic hardware, a promising technology to speed up
artificial neural networks, as well as of biological networks. We show that our
theoretical bounds can be used to compare the fault tolerance of different
architectures and to design a regularizer improving the fault tolerance of a
given architecture. We design an algorithm achieving fault tolerance using a
reasonable number of neurons. In addition to the theoretical proof, we also
provide experimental validation of our results and suggest a connection to the
generalization capacity problem.
