Tiny Transformers for Environmental Sound Classification at the Edge
  With the growth of the Internet of Things and the rise of Big Data, data
processing and machine learning applications are being moved to cheap and low
size, weight, and power (SWaP) devices at the edge, often in the form of mobile
phones, embedded systems, or microcontrollers. The field of Cyber-Physical
Measurements and Signature Intelligence (MASINT) makes use of these devices to
analyze and exploit data in ways not otherwise possible, which results in
increased data quality, increased security, and decreased bandwidth. However,
methods to train and deploy models at the edge are limited, and models with
sufficient accuracy are often too large for the edge device. Therefore, there
is a clear need for techniques to create efficient AI/ML at the edge. This work
presents training techniques for audio models in the field of environmental
sound classification at the edge. Specifically, we design and train
Transformers to classify office sounds in audio clips. Results show that a
BERT-based Transformer, trained on Mel spectrograms, can outperform a CNN using
99.85% fewer parameters. To achieve this result, we first tested several audio
feature extraction techniques designed for Transformers, using ESC-50 for
evaluation, along with various augmentations. Our final model outperforms the
state-of-the-art MFCC-based CNN on the office sounds dataset, using just over
6,000 parameters -- small enough to run on a microcontroller.
