ReLU Deep Neural Networks from the Hierarchical Basis Perspective
  We study ReLU deep neural networks (DNNs) by investigating their connections
with the hierarchical basis method in finite element methods. First, we show
that the approximation schemes of ReLU DNNs for $x^2$ and $xy$ are composition
versions of the hierarchical basis approximation for these two functions. Based
on this fact, we obtain a geometric interpretation and systematic proof for the
approximation result of ReLU DNNs for polynomials, which plays an important
role in a series of recent exponential approximation results of ReLU DNNs.
Through our investigation of connections between ReLU DNNs and the hierarchical
basis approximation for $x^2$ and $xy$, we show that ReLU DNNs with this
special structure can be applied only to approximate quadratic functions.
Furthermore, we obtain a concise representation to explicitly reproduce any
linear finite element function on a two-dimensional uniform mesh by using ReLU
DNNs with only two hidden layers.
