Information theory and learning: a physical approach
  We try to establish a unified information theoretic approach to learning and
to explore some of its applications. First, we define {\em predictive
information} as the mutual information between the past and the future of a
time series, discuss its behavior as a function of the length of the series,
and explain how other quantities of interest studied previously in learning
theory - as well as in dynamical systems and statistical mechanics - emerge
from this universally definable concept. We then prove that predictive
information provides the {\em unique measure for the complexity} of dynamics
underlying the time series and show that there are classes of models
characterized by {\em power-law growth of the predictive information} that are
qualitatively more complex than any of the systems that have been investigated
before. Further, we investigate numerically the learning of a nonparametric
probability density, which is an example of a problem with power-law
complexity, and show that the proper Bayesian formulation of this problem
provides for the `Occam' factors that punish overly complex models and thus
allow one {\em to learn not only a solution within a specific model class, but
also the class itself} using the data only and with very few a priori
assumptions. We study a possible {\em information theoretic method} that
regularizes the learning of an undersampled discrete variable, and show that
learning in such a setup goes through stages of very different complexities.
Finally, we discuss how all of these ideas may be useful in various problems in
physics, statistics, and, most importantly, biology.
