Reconstruction of training samples from loss functions
  This paper presents a new mathematical framework to analyze the loss
functions of deep neural networks with ReLU functions. Furthermore, as as
application of this theory, we prove that the loss functions can reconstruct
the inputs of the training samples up to scalar multiplication (as vectors) and
can provide the number of layers and nodes of the deep neural network. Namely,
if we have all input and output of a loss function (or equivalently all
possible learning process), for all input of each training sample $x_i \in
\mathbb{R}^n$, we can obtain vectors $x'_i\in \mathbb{R}^n$ satisfying
$x_i=c_ix'_i$ for some $c_i \neq 0$. To prove theorem, we introduce the notion
of virtual polynomials, which are polynomials written as the output of a node
in a deep neural network. Using virtual polynomials, we find an algebraic
structure for the loss surfaces, called semi-algebraic sets. We analyze these
loss surfaces from the algebro-geometric point of view. Factorization of
polynomials is one of the most standard ideas in algebra. Hence, we express the
factorization of the virtual polynomials in terms of their active paths. This
framework can be applied to the leakage problem in the training of deep neural
networks. The main theorem in this paper indicates that there are many risks
associated with the training of deep neural networks. For example, if we have N
(the dimension of weight space) + 1 nonsmooth points on the loss surface, which
are sufficiently close to each other, we can obtain the input of training
sample up to scalar multiplication. We also point out that the structures of
the loss surfaces depend on the shape of the deep neural network and not on the
training samples.
