The Spectral Bias of Polynomial Neural Networks
  Polynomial neural networks (PNNs) have been recently shown to be particularly
effective at image generation and face recognition, where high-frequency
information is critical. Previous studies have revealed that neural networks
demonstrate a $\textit{spectral bias}$ towards low-frequency functions, which
yields faster learning of low-frequency components during training. Inspired by
such studies, we conduct a spectral analysis of the Neural Tangent Kernel (NTK)
of PNNs. We find that the $\Pi$-Net family, i.e., a recently proposed
parametrization of PNNs, speeds up the learning of the higher frequencies. We
verify the theoretical bias through extensive experiments. We expect our
analysis to provide novel insights into designing architectures and learning
frameworks by incorporating multiplicative interactions via polynomials.
