Learning neural audio features without supervision
  Deep audio classification, traditionally cast as training a deep neural
network on top of mel-filterbanks in a supervised fashion, has recently
benefited from two independent lines of work. The first one explores "learnable
frontends", i.e., neural modules that produce a learnable time-frequency
representation, to overcome limitations of fixed features. The second one uses
self-supervised learning to leverage unprecedented scales of pre-training data.
In this work, we study the feasibility of combining both approaches, i.e.,
pre-training learnable frontend jointly with the main architecture for
downstream classification. First, we show that pretraining two previously
proposed frontends (SincNet and LEAF) on Audioset drastically improves
linear-probe performance over fixed mel-filterbanks, suggesting that learnable
time-frequency representations can benefit self-supervised pre-training even
more than supervised training. Surprisingly, randomly initialized learnable
filterbanks outperform mel-scaled initialization in the self-supervised
setting, a counter-intuitive result that questions the appropriateness of
strong priors when designing learnable filters. Through exploratory analysis of
the learned frontend components, we uncover crucial differences in properties
of these frontends when used in a supervised and self-supervised setting,
especially the affinity of self-supervised filters to diverge significantly
from the mel-scale to model a broader range of frequencies.
