Study of positional encoding approaches for Audio Spectrogram
  Transformers
  Transformers have revolutionized the world of deep learning, specially in the
field of natural language processing. Recently, the Audio Spectrogram
Transformer (AST) was proposed for audio classification, leading to state of
the art results in several datasets. However, in order for ASTs to outperform
CNNs, pretraining with ImageNet is needed. In this paper, we study one
component of the AST, the positional encoding, and propose several variants to
improve the performance of ASTs trained from scratch, without ImageNet
pretraining. Our best model, which incorporates conditional positional
encodings, significantly improves performance on Audioset and ESC-50 compared
to the original AST.
