Conformer CTC dropping spoken numbers
We have successfully trained a Conformer CTC BPE L model (non-English) with a very good WER in the lower teens mark. There is one peculiar thing that is happening though. The model is dropping numbers, interestingly not always and mostly dates. Especially if more are spoken in succession as part of a sentence. I have found a similar issue/questionon regarding the Spanish language on Riva Forums https://forums.developer.nvidia.com/t/riva-with-asr-doesnt-recognize-numbers-on-my-audio/203244, but no explanation. 

It is true that the train dataset does not cover such specific cases, but we were under the impression that in general the model learns to recognise BPE tokens from the waveform, and no "internal LM" or context dependency between BPE tokens (or at max a short one). In any case we'd expect the model to output at least something, not miss entire sections with multiple consecutive words. An example of the worst case we are seeing (translated in English) would be:
- reference: "accepted on twelfth fifth twenty twenty and modified on seventh seventh twenty twenty-one"
- transcription: "accepted and modified"  

We also get different results depending if we run transcription with `transcribe_speech.py`, `transcribe_speech_parallel.py` or buffered inference. 

Any thoughts to what might be the source of the problem, or what can we do to resolve it?

