Analytical aspects of non-differentiable neural networks
  Research in computational deep learning has directed considerable efforts
towards hardware-oriented optimisations for deep neural networks, via the
simplification of the activation functions, or the quantization of both
activations and weights. The resulting non-differentiability (or even
discontinuity) of the networks poses some challenging problems, especially in
connection with the learning process. In this paper, we address several
questions regarding both the expressivity of quantized neural networks and
approximation techniques for non-differentiable networks. First, we answer in
the affirmative the question of whether QNNs have the same expressivity as DNNs
in terms of approximation of Lipschitz functions in the $L^{\infty}$ norm.
Then, considering a continuous but not necessarily differentiable network, we
describe a layer-wise stochastic regularisation technique to produce
differentiable approximations, and we show how this approach to regularisation
provides elegant quantitative estimates. Finally, we consider networks defined
by means of Heaviside-type activation functions, and prove for them a pointwise
approximation result by means of smooth networks under suitable assumptions on
the regularised activations.
