Grassmannian Packings in Neural Networks: Learning with Maximal Subspace
  Packings for Diversity and Anti-Sparsity
  Kernel sparsity ("dying ReLUs") and lack of diversity are commonly observed
in CNN kernels, which decreases model capacity. Drawing inspiration from
information theory and wireless communications, we demonstrate the intersection
of coding theory and deep learning through the Grassmannian subspace packing
problem in CNNs. We propose Grassmannian packings for initial kernel layers to
be initialized maximally far apart based on chordal or Fubini-Study distance.
Convolutional kernels initialized with Grassmannian packings exhibit diverse
features and obtain diverse representations. We show that Grassmannian
packings, especially in the initial layers, address kernel sparsity and
encourage diversity, while improving classification accuracy across shallow and
deep CNNs with better convergence rates.
