On Theory-training Neural Networks to Infer the Solution of Highly
  Coupled Differential Equations
  Deep neural networks are transforming fields ranging from computer vision to
computational medicine, and we recently extended their application to the field
of phase-change heat transfer by introducing theory-trained neural networks
(TTNs) for a solidification problem \cite{TTN}. Here, we present general,
in-depth, and empirical insights into theory-training networks for learning the
solution of highly coupled differential equations. We analyze the deteriorating
effects of the oscillating loss on the ability of a network to satisfy the
equations at the training data points, measured by the final training loss, and
on the accuracy of the inferred solution. We introduce a theory-training
technique that, by leveraging regularization, eliminates those oscillations,
decreases the final training loss, and improves the accuracy of the inferred
solution, with no additional computational cost. Then, we present guidelines
that allow a systematic search for the network that has the optimal training
time and inference accuracy for a given set of equations; following these
guidelines can reduce the number of tedious training iterations in that search.
Finally, a comparison between theory-training and the rival, conventional
method of solving differential equations using discretization attests to the
advantages of theory-training not being necessarily limited to high-dimensional
sets of equations. The comparison also reveals a limitation of the current
theory-training framework that may limit its application in domains where
extreme accuracies are necessary.
