A Deep Neural Network's Loss Surface Contains Every Low-dimensional
  Pattern
  The work "Loss Landscape Sightseeing with Multi-Point Optimization"
(Skorokhodov and Burtsev, 2019) demonstrated that one can empirically find
arbitrary 2D binary patterns inside loss surfaces of popular neural networks.
In this paper we prove that: (i) this is a general property of deep universal
approximators; and (ii) this property holds for arbitrary smooth patterns, for
other dimensionalities, for every dataset, and any neural network that is
sufficiently deep and wide. Our analysis predicts not only the existence of all
such low-dimensional patterns, but also two other properties that were observed
empirically: (i) that it is easy to find these patterns; and (ii) that they
transfer to other data-sets (e.g. a test-set).
