Depth separation for reduced deep networks in nonlinear model reduction:
  Distilling shock waves in nonlinear hyperbolic problems
  Classical reduced models are low-rank approximations using a fixed basis
designed to achieve dimensionality reduction of large-scale systems. In this
work, we introduce reduced deep networks, a generalization of classical reduced
models formulated as deep neural networks. We prove depth separation results
showing that reduced deep networks approximate solutions of parametrized
hyperbolic partial differential equations with approximation error $\epsilon$
with $\mathcal{O}(|\log(\epsilon)|)$ degrees of freedom, even in the nonlinear
setting where solutions exhibit shock waves. We also show that classical
reduced models achieve exponentially worse approximation rates by establishing
lower bounds on the relevant Kolmogorov $N$-widths.
