Are discrete units necessary for Spoken Language Modeling?
  Recent work in spoken language modeling shows the possibility of learning a
language unsupervisedly from raw audio without any text labels. The approach
relies first on transforming the audio into a sequence of discrete units (or
pseudo-text) and then training a language model directly on such pseudo-text.
Is such a discrete bottleneck necessary, potentially introducing irreversible
errors in the encoding of the speech signal, or could we learn a language model
without discrete units at all? In this work, show that discretization is indeed
essential for good results in spoken language modeling, but that can omit the
discrete bottleneck if we use using discrete target features from a higher
level than the input features. We also show that an end-to-end model trained
with discrete target like HuBERT achieves similar results as the best language
model trained on pseudo-text on a set of zero-shot spoken language modeling
metrics from the Zero Resource Speech Challenge 2021.
