Universal approximation and model compression for radial neural networks
  We introduce a class of fully-connected neural networks whose activation
functions, rather than being pointwise, rescale feature vectors by a function
depending only on their norm. We call such networks radial neural networks,
extending previous work on rotation equivariant networks that considers
rescaling activations in less generality. We prove universal approximation
theorems for radial neural networks, including in the more difficult cases of
bounded widths and unbounded domains. Our proof techniques are novel, distinct
from those in the pointwise case. Additionally, radial neural networks exhibit
a rich group of orthogonal change-of-basis symmetries on the vector space of
trainable parameters. Factoring out these symmetries leads to a practical
lossless model compression algorithm. Optimization of the compressed model by
gradient descent is equivalent to projected gradient descent for the full
model.
