Integration of Neural Network-Based Symbolic Regression in Deep Learning
  for Scientific Discovery
  Symbolic regression is a powerful technique that can discover analytical
equations that describe data, which can lead to explainable models and
generalizability outside of the training data set. In contrast, neural networks
have achieved amazing levels of accuracy on image recognition and natural
language processing tasks, but are often seen as black-box models that are
difficult to interpret and typically extrapolate poorly. Here we use a neural
network-based architecture for symbolic regression called the Equation Learner
(EQL) network and integrate it with other deep learning architectures such that
the whole system can be trained end-to-end through backpropagation. To
demonstrate the power of such systems, we study their performance on several
substantially different tasks. First, we show that the neural network can
perform symbolic regression and learn the form of several functions. Next, we
present an MNIST arithmetic task where a separate part of the neural network
extracts the digits. Finally, we demonstrate prediction of dynamical systems
where an unknown parameter is extracted through an encoder. We find that the
EQL-based architecture can extrapolate quite well outside of the training data
set compared to a standard neural network-based architecture, paving the way
for deep learning to be applied in scientific exploration and discovery.
