[BUG] DALI result is worse than normal pytorch data loader
Hello,

When running the same training or even the same validation script without DALI, we get the expected WER.
However, when we enable DALI using the same configuration, we get much worse WER.

I have created a Google Colab example that replicates this issue, which can be found here:
https://colab.research.google.com/drive/1Rz42EeQVDHhTso3kWvBp1wQc6x1FxXgY?usp=sharing

To make the debugging more straightforward, I have disabled the training on the example.
So simply, it performs testing with DALI and without DALI.

This is related to issue #2817 , which was fixed by @titu1994 .