On the eigenvector bias of Fourier feature networks: From regression to
  solving multi-scale PDEs with physics-informed neural networks
  Physics-informed neural networks (PINNs) are demonstrating remarkable promise
in integrating physical models with gappy and noisy observational data, but
they still struggle in cases where the target functions to be approximated
exhibit high-frequency or multi-scale features. In this work we investigate
this limitation through the lens of Neural Tangent Kernel (NTK) theory and
elucidate how PINNs are biased towards learning functions along the dominant
eigen-directions of their limiting NTK. Using this observation, we construct
novel architectures that employ spatio-temporal and multi-scale random Fourier
features, and justify how such coordinate embedding layers can lead to robust
and accurate PINN models. Numerical examples are presented for several
challenging cases where conventional PINN models fail, including wave
propagation and reaction-diffusion dynamics, illustrating how the proposed
methods can be used to effectively tackle both forward and inverse problems
involving partial differential equations with multi-scale behavior. All code an
data accompanying this manuscript will be made publicly available at
\url{https://github.com/PredictiveIntelligenceLab/MultiscalePINNs}.
