Measuring the Intrinsic Dimension of Objective Landscapes
  Many recently trained neural networks employ large numbers of parameters to
achieve good performance. One may intuitively use the number of parameters
required as a rough gauge of the difficulty of a problem. But how accurate are
such notions? How many parameters are really needed? In this paper we attempt
to answer this question by training networks not in their native parameter
space, but instead in a smaller, randomly oriented subspace. We slowly increase
the dimension of this subspace, note at which dimension solutions first appear,
and define this to be the intrinsic dimension of the objective landscape. The
approach is simple to implement, computationally tractable, and produces
several suggestive conclusions. Many problems have smaller intrinsic dimensions
than one might suspect, and the intrinsic dimension for a given dataset varies
little across a family of models with vastly different sizes. This latter
result has the profound implication that once a parameter space is large enough
to solve a problem, extra parameters serve directly to increase the
dimensionality of the solution manifold. Intrinsic dimension allows some
quantitative comparison of problem difficulty across supervised, reinforcement,
and other types of learning where we conclude, for example, that solving the
inverted pendulum problem is 100 times easier than classifying digits from
MNIST, and playing Atari Pong from pixels is about as hard as classifying
CIFAR-10. In addition to providing new cartography of the objective landscapes
wandered by parameterized models, the method is a simple technique for
constructively obtaining an upper bound on the minimum description length of a
solution. A byproduct of this construction is a simple approach for compressing
networks, in some cases by more than 100 times.
