Reduced-Order Neural Network Synthesis with Robustness Guarantees
  In the wake of the explosive growth in smartphones and cyberphysical systems,
there has been an accelerating shift in how data is generated away from
centralised data towards on-device generated data. In response, machine
learning algorithms are being adapted to run locally on board, potentially
hardware limited, devices to improve user privacy, reduce latency and be more
energy efficient. However, our understanding of how these device orientated
algorithms behave and should be trained is still fairly limited. To address
this issue, a method to automatically synthesize reduced-order neural networks
(having fewer neurons) approximating the input/output mapping of a larger one
is introduced. The reduced-order neural network's weights and biases are
generated from a convex semi-definite programme that minimises the worst-case
approximation error with respect to the larger network. Worst-case bounds for
this approximation error are obtained and the approach can be applied to a wide
variety of neural networks architectures. What differentiates the proposed
approach to existing methods for generating small neural networks, e.g.
pruning, is the inclusion of the worst-case approximation error directly within
the training cost function, which should add robustness. Numerical examples
highlight the potential of the proposed approach. The overriding goal of this
paper is to generalise recent results in the robustness analysis of neural
networks to a robust synthesis problem for their weights and biases.
