Statistical-mechanical analysis of pre-training and fine tuning in deep
  learning
  In this paper, we present a statistical-mechanical analysis of deep learning.
We elucidate some of the essential components of deep learning---pre-training
by unsupervised learning and fine tuning by supervised learning. We formulate
the extraction of features from the training data as a margin criterion in a
high-dimensional feature-vector space. The self-organized classifier is then
supplied with small amounts of labelled data, as in deep learning. Although we
employ a simple single-layer perceptron model, rather than directly analyzing a
multi-layer neural network, we find a nontrivial phase transition that is
dependent on the number of unlabelled data in the generalization error of the
resultant classifier. In this sense, we evaluate the efficacy of the
unsupervised learning component of deep learning. The analysis is performed by
the replica method, which is a sophisticated tool in statistical mechanics. We
validate our result in the manner of deep learning, using a simple iterative
algorithm to learn the weight vector on the basis of belief propagation.
