Pixelated Butterfly: Simple and Efficient Sparse training for Neural
  Network Models
  Overparameterized neural networks generalize well but are expensive to train.
Ideally, one would like to reduce their computational cost while retaining
their generalization benefits. Sparse model training is a simple and promising
approach to achieve this, but there remain challenges as existing methods
struggle with accuracy loss, slow training runtime, or difficulty in
sparsifying all model components. The core problem is that searching for a
sparsity mask over a discrete set of sparse matrices is difficult and
expensive. To address this, our main insight is to optimize over a continuous
superset of sparse matrices with a fixed structure known as products of
butterfly matrices. As butterfly matrices are not hardware efficient, we
propose simple variants of butterfly (block and flat) to take advantage of
modern hardware. Our method (Pixelated Butterfly) uses a simple fixed sparsity
pattern based on flat block butterfly and low-rank matrices to sparsify most
network layers (e.g., attention, MLP). We empirically validate that Pixelated
Butterfly is 3x faster than butterfly and speeds up training to achieve
favorable accuracy--efficiency tradeoffs. On the ImageNet classification and
WikiText-103 language modeling tasks, our sparse models train up to 2.5x faster
than the dense MLP-Mixer, Vision Transformer, and GPT-2 medium with no drop in
accuracy.
