How to inference using fp16 with a fp32 trained ASR model?
You can do it like this with autocast of PT:

```
  with torch.cuda.amp.autocast():
      with torch.no_grad():
          transcriptions = asr_model.transcribe(filepaths, batch_size=cfg.batch_size)

``` @VahidooX this answer is correct. You can use the pytorch cuda decorator for AMP as above to perform FP16 inference on GPU Does not work for https://huggingface.co/nvidia/parakeet-rnnt-0.6b
I trained in FP32 on RTX titan, infer on fp16 give all `??` and is taking 5 times longer. FP32 works fine.

Noteworthy is that training with fp16 on this model also does not work. @nithinraok, have we tried this model with fp16?

There are cases where a model trained in fp32 does not work with fp16 as some weights are out of the range or activations get too large in some layers. Yeah these models are trained with bf16 and haven't tested or updated for fp16. Works out of the box with either fp32 or bf16. 