The gap between theory and practice in function approximation with deep
  neural networks
  Deep learning (DL) is transforming industry as decision-making processes are
being automated by deep neural networks (DNNs) trained on real-world data.
Driven partly by rapidly-expanding literature on DNN approximation theory
showing they can approximate a rich variety of functions, such tools are
increasingly being considered for problems in scientific computing. Yet, unlike
traditional algorithms in this field, little is known about DNNs from the
principles of numerical analysis, e.g., stability, accuracy, computational
efficiency and sample complexity. In this paper we introduce a computational
framework for examining DNNs in practice, and use it to study empirical
performance with regard to these issues. We study performance of DNNs of
different widths & depths on test functions in various dimensions, including
smooth and piecewise smooth functions. We also compare DL against best-in-class
methods for smooth function approx. based on compressed sensing (CS). Our main
conclusion from these experiments is that there is a crucial gap between the
approximation theory of DNNs and their practical performance, with trained DNNs
performing relatively poorly on functions for which there are strong
approximation results (e.g. smooth functions), yet performing well in
comparison to best-in-class methods for other functions. To analyze this gap
further, we provide some theoretical insights. We establish a practical
existence theorem, asserting existence of a DNN architecture and training
procedure that offers the same performance as CS. This establishes a key
theoretical benchmark, showing the gap can be closed, albeit via a strategy
guaranteed to perform as well as, but no better than, current best-in-class
schemes. Nevertheless, it demonstrates the promise of practical DNN approx., by
highlighting potential for better schemes through careful design of DNN
architectures and training strategies.
