TinySpeech: Attention Condensers for Deep Speech Recognition Neural
  Networks on Edge Devices
  Advances in deep learning have led to state-of-the-art performance across a
multitude of speech recognition tasks. Nevertheless, the widespread deployment
of deep neural networks for on-device speech recognition remains a challenge,
particularly in edge scenarios where the memory and computing resources are
highly constrained (e.g., low-power embedded devices) or where the memory and
computing budget dedicated to speech recognition is low (e.g., mobile devices
performing numerous tasks besides speech recognition). In this study, we
introduce the concept of attention condensers for building low-footprint,
highly-efficient deep neural networks for on-device speech recognition on the
edge. An attention condenser is a self-attention mechanism that learns and
produces a condensed embedding characterizing joint local and cross-channel
activation relationships, and performs selective attention accordingly. To
illustrate its efficacy, we introduce TinySpeech, low-precision deep neural
networks comprising largely of attention condensers tailored for on-device
speech recognition using a machine-driven design exploration strategy, with one
tailored specifically with microcontroller operation constraints. Experimental
results on the Google Speech Commands benchmark dataset for limited-vocabulary
speech recognition showed that TinySpeech networks achieved significantly lower
architectural complexity (as much as $507\times$ fewer parameters), lower
computational complexity (as much as $48\times$ fewer multiply-add operations),
and lower storage requirements (as much as $2028\times$ lower weight memory
requirements) when compared to previous work. These results not only
demonstrate the efficacy of attention condensers for building highly efficient
networks for on-device speech recognition, but also illuminate its potential
for accelerating deep learning on the edge and empowering TinyML applications.
