Algorithmic Probability of Large Datasets and the Simplicity Bubble
  Problem in Machine Learning
  When mining large datasets in order to predict new data, limitations of the
principles behind statistical machine learning pose a serious challenge not
only to the Big Data deluge, but also to the traditional assumptions that data
generating processes are biased toward low algorithmic complexity. Even when
one assumes an underlying algorithmic-informational bias toward simplicity in
finite dataset generators, we show that fully automated, with or without access
to pseudo-random generators, computable learning algorithms, in particular
those of statistical nature used in current approaches to machine learning
(including deep learning), can always be deceived, naturally or artificially,
by sufficiently large datasets. In particular, we demonstrate that, for every
finite learning algorithm, there is a sufficiently large dataset size above
which the algorithmic probability of an unpredictable deceiver is an upper
bound (up to a multiplicative constant that only depends on the learning
algorithm) for the algorithmic probability of any other larger dataset. In
other words, very large and complex datasets are as likely to deceive learning
algorithms into a "simplicity bubble" as any other particular dataset. These
deceiving datasets guarantee that any prediction will diverge from the
high-algorithmic-complexity globally optimal solution while converging toward
the low-algorithmic-complexity locally optimal solution. We discuss the
framework and empirical conditions for circumventing this deceptive phenomenon,
moving away from statistical machine learning towards a stronger type of
machine learning based on, or motivated by, the intrinsic power of algorithmic
information theory and computability theory.
