Learning to Count Words in Fluent Speech enables Online Speech
  Recognition
  Sequence to Sequence models, in particular the Transformer, achieve state of
the art results in Automatic Speech Recognition. Practical usage is however
limited to cases where full utterance latency is acceptable. In this work we
introduce Taris, a Transformer-based online speech recognition system aided by
an auxiliary task of incremental word counting. We use the cumulative word sum
to dynamically segment speech and enable its eager decoding into words.
Experiments performed on the LRS2, LibriSpeech, and Aishell-1 datasets of
English and Mandarin speech show that the online system performs comparable
with the offline one when having a dynamic algorithmic delay of 5 segments.
Furthermore, we show that the estimated segment length distribution resembles
the word length distribution obtained with forced alignment, although our
system does not require an exact segment-to-word equivalence. Taris introduces
a negligible overhead compared to a standard Transformer, while the local
relationship modelling between inputs and outputs grants invariance to sequence
length by design.
