On Compression Principle and Bayesian Optimization for Neural Networks
  Finding methods for making generalizable predictions is a fundamental problem
of machine learning. By looking into similarities between the prediction
problem for unknown data and the lossless compression we have found an approach
that gives a solution. In this paper we propose a compression principle that
states that an optimal predictive model is the one that minimizes a total
compressed message length of all data and model definition while guarantees
decodability. Following the compression principle we use Bayesian approach to
build probabilistic models of data and network definitions. A method to
approximate Bayesian integrals using a sequence of variational approximations
is implemented as an optimizer for hyper-parameters: Bayesian Stochastic
Gradient Descent (BSGD). Training with BSGD is completely defined by setting
only three parameters: number of epochs, the size of the dataset and the size
of the minibatch, which define a learning rate and a number of iterations. We
show that dropout can be used for a continuous dimensionality reduction that
allows to find optimal network dimensions as required by the compression
principle.
