Expressive Power and Loss Surfaces of Deep Learning Models
  The goals of this paper are two-fold. The first goal is to serve as an
expository tutorial on the working of deep learning models which emphasizes
geometrical intuition about the reasons for success of deep learning. The
second goal is to complement the current results on the expressive power of
deep learning models and their loss surfaces with novel insights and results.
In particular, we describe how deep neural networks carve out manifolds
especially when the multiplication neurons are introduced. Multiplication is
used in dot products and the attention mechanism and it is employed in capsule
networks and self-attention based transformers. We also describe how random
polynomial, random matrix, spin glass and computational complexity perspectives
on the loss surfaces are interconnected.
