Self-scalable Tanh (Stan): Faster Convergence and Better Generalization
  in Physics-informed Neural Networks
  Physics-informed Neural Networks (PINNs) are gaining attention in the
engineering and scientific literature for solving a range of differential
equations with applications in weather modeling, healthcare, manufacturing,
etc. Poor scalability is one of the barriers to utilizing PINNs for many
real-world problems. To address this, a Self-scalable tanh (Stan) activation
function is proposed for the PINNs. The proposed Stan function is smooth,
non-saturating, and has a trainable parameter. During training, it can allow
easy flow of gradients to compute the required derivatives and also enable
systematic scaling of the input-output mapping. It is shown theoretically that
the PINNs with the proposed Stan function have no spurious stationary points
when using gradient descent algorithms. The proposed Stan is tested on a number
of numerical studies involving general regression problems. It is subsequently
used for solving multiple forward problems, which involve second-order
derivatives and multiple dimensions, and an inverse problem where the thermal
diffusivity of a rod is predicted with heat conduction data. These case studies
establish empirically that the Stan activation function can achieve better
training and more accurate predictions than the existing activation functions
in the literature.
