Conformer: Convolution-augmented Transformer for Speech Recognition
  Recently Transformer and Convolution neural network (CNN) based models have
shown promising results in Automatic Speech Recognition (ASR), outperforming
Recurrent neural networks (RNNs). Transformer models are good at capturing
content-based global interactions, while CNNs exploit local features
effectively. In this work, we achieve the best of both worlds by studying how
to combine convolution neural networks and transformers to model both local and
global dependencies of an audio sequence in a parameter-efficient way. To this
regard, we propose the convolution-augmented transformer for speech
recognition, named Conformer. Conformer significantly outperforms the previous
Transformer and CNN based models achieving state-of-the-art accuracies. On the
widely used LibriSpeech benchmark, our model achieves WER of 2.1%/4.3% without
using a language model and 1.9%/3.9% with an external language model on
test/testother. We also observe competitive performance of 2.7%/6.3% with a
small model of only 10M parameters.
