nemo2riva of tts_en_fastpitch:1.4.0 fails with CUDA out of memory
@borisfom As per comment https://github.com/NVIDIA/NeMo/issues/3619#issuecomment-1089551439, I'm re-opening the bug as the issue persists with `nemo:22.01` (1.7.1), `nemo:1.7.2` and `nemo:1.9.0rc0` (e4ee26b601a8e6f907f926019367d3d545c62f11) combined with `Riva:2.0.0`. I tried both by starting off with `nemo:22.01` and upgrading to more recent versions of nemo, as well as building new containers from the specific GitHub repo tag. I tested with T4 (16GB) and A100 (40GB) GPUs. On all occasions the conversion of [tts_en_fastpitch_align.nemo](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/tts_en_fastpitch) fails with `RuntimeError: CUDA out of memory`. 