Approximation with Tensor Networks. Part I: Approximation Spaces
  We study the approximation of functions by tensor networks (TNs). We show
that Lebesgue $L^p$-spaces in one dimension can be identified with tensor
product spaces of arbitrary order through tensorization. We use this tensor
product structure to define subsets of $L^p$ of rank-structured functions of
finite representation complexity. These subsets are then used to define
different approximation classes of tensor networks, associated with different
measures of complexity. These approximation classes are shown to be
quasi-normed linear spaces. We study some elementary properties and
relationships of said spaces. In part II of this work, we will show that
classical smoothness (Besov) spaces are continuously embedded into these
approximation classes. We will also show that functions in these approximation
classes do not possess any Besov smoothness, unless one restricts the depth of
the tensor networks. The results of this work are both an analysis of the
approximation spaces of TNs and a study of the expressivity of a particular
type of neural networks (NN) -- namely feed-forward sum-product networks with
sparse architecture. The input variables of this network result from the
tensorization step, interpreted as a particular featuring step which can also
be implemented with a neural network with a specific architecture. We point out
interesting parallels to recent results on the expressivity of rectified linear
unit (ReLU) networks -- currently one of the most popular type of NNs.
