SpecTNT: a Time-Frequency Transformer for Music Audio
  Transformers have drawn attention in the MIR field for their remarkable
performance shown in natural language processing and computer vision. However,
prior works in the audio processing domain mostly use Transformer as a temporal
feature aggregator that acts similar to RNNs. In this paper, we propose
SpecTNT, a Transformer-based architecture to model both spectral and temporal
sequences of an input time-frequency representation. Specifically, we introduce
a novel variant of the Transformer-in-Transformer (TNT) architecture. In each
SpecTNT block, a spectral Transformer extracts frequency-related features into
the frequency class token (FCT) for each frame. Later, the FCTs are linearly
projected and added to the temporal embeddings (TEs), which aggregate useful
information from the FCTs. Then, a temporal Transformer processes the TEs to
exchange information across the time axis. By stacking the SpecTNT blocks, we
build the SpecTNT model to learn the representation for music signals. In
experiments, SpecTNT demonstrates state-of-the-art performance in music tagging
and vocal melody extraction, and shows competitive performance for chord
recognition. The effectiveness of SpecTNT and other design choices are further
examined through ablation studies.
