Fast Neural Models for Symbolic Regression at Scale
  Deep learning owes much of its success to the astonishing expressiveness of
neural networks. However, this comes at the cost of complex, black-boxed models
that extrapolate poorly beyond the domain of the training dataset, conflicting
with goals of finding analytic expressions to describe science, engineering and
real world data. Under the hypothesis that the hierarchical modularity of such
laws can be captured by training a neural network, we introduce OccamNet, a
neural network model that finds interpretable, compact, and sparse solutions
for fitting data, \`{a} la Occam's razor. Our model defines a probability
distribution over a non-differentiable function space. We introduce a two-step
optimization method that samples functions and updates the weights with
backpropagation based on cross-entropy matching in an evolutionary strategy: we
train by biasing the probability mass toward better fitting solutions. OccamNet
is able to fit a variety of symbolic laws including simple analytic functions,
recursive programs, implicit functions, simple image classification, and can
outperform noticeably state-of-the-art symbolic regression methods on real
world regression datasets. Our method requires minimal memory footprint, does
not require AI accelerators for efficient training, fits complicated functions
in minutes of training on a single CPU, and demonstrates significant
performance gains when scaled on a GPU. Our implementation, demonstrations and
instructions for reproducing the experiments are available at
https://github.com/druidowm/OccamNet_Public.
