Trace norm regularization and faster inference for embedded speech
  recognition RNNs
  We propose and evaluate new techniques for compressing and speeding up dense
matrix multiplications as found in the fully connected and recurrent layers of
neural networks for embedded large vocabulary continuous speech recognition
(LVCSR). For compression, we introduce and study a trace norm regularization
technique for training low rank factored versions of matrix multiplications.
Compared to standard low rank training, we show that our method leads to good
accuracy versus number of parameter trade-offs and can be used to speed up
training of large models. For speedup, we enable faster inference on ARM
processors through new open sourced kernels optimized for small batch sizes,
resulting in 3x to 7x speed ups over the widely used gemmlowp library. Beyond
LVCSR, we expect our techniques and kernels to be more generally applicable to
embedded neural networks with large fully connected or recurrent layers.
