Analytically Embedding Differential Equation Constraints into Least
  Squares Support Vector Machines using the Theory of Functional Connections
  Differential equations (DEs) are used as numerical models to describe
physical phenomena throughout the field of engineering and science, including
heat and fluid flow, structural bending, and systems dynamics. While there are
many other techniques for finding approximate solutions to these equations,
this paper looks to compare the application of the Theory of Functional
Connections (TFC) with one based on least-squares support vector machines
(LS-SVM). The TFC method uses a constrained expression, an expression that
always satisfies the DE constraints, which transforms the process of solving a
DE into solving an unconstrained optimization problem that is ultimately solved
via least-squares (LS). In addition to individual analysis, the two methods are
merged into a new methodology, called constrained SVMs (CSVM), by incorporating
the LS-SVM method into the TFC framework to solve unconstrained problems.
Numerical tests are conducted on four sample problems: One first order linear
ordinary differential equation (ODE), one first order nonlinear ODE, one second
order linear ODE, and one two-dimensional linear partial differential equation
(PDE). Using the LS-SVM method as a benchmark, a speed comparison is made for
all the problems by timing the training period, and an accuracy comparison is
made using the maximum error and mean squared error on the training and test
sets. In general, TFC is shown to be slightly faster (by an order of magnitude
or less) and more accurate (by multiple orders of magnitude) than the LS-SVM
and CSVM approaches.
