Symmetry-via-Duality: Invariant Neural Network Densities from
  Parameter-Space Correlators
  Parameter-space and function-space provide two different duality frames in
which to study neural networks. We demonstrate that symmetries of network
densities may be determined via dual computations of network correlation
functions, even when the density is unknown and the network is not equivariant.
Symmetry-via-duality relies on invariance properties of the correlation
functions, which stem from the choice of network parameter distributions. Input
and output symmetries of neural network densities are determined, which recover
known Gaussian process results in the infinite width limit. The mechanism may
also be utilized to determine symmetries during training, when parameters are
correlated, as well as symmetries of the Neural Tangent Kernel. We demonstrate
that the amount of symmetry in the initialization density affects the accuracy
of networks trained on Fashion-MNIST, and that symmetry breaking helps only
when it is in the direction of ground truth.
