High GPU usage in Google Colab
**Description**

While using NeMo in google golab in CPU mode the GPU spike to 100% never comes down. the same issue is in the TPU mode Also the RAM spiked to 100%

**Steps to reproduce bug**
Try to run this collab file, the same issue will be seen
````
https://colab.research.google.com/drive/1uO2cgu5wjZqEUY1wnlHCe0npe3XRDTka?usp=sharing
````

**Expected behavior**

A clear and concise description of what you expected to happen.

 - Environment Location:  Collab
 - Method of NeMo install: While running this code 

````
parsed = spec_generator.parse(input_text)
spectrogram = spec_generator.generate_spectrogram(tokens=parsed)
audio = model.convert_spectrogram_to_audio(spec=spectrogram)
````
**Environment details**

Otherwise, please provide:
- OS version
- PyTorch Version
````
pytorch-lightning==1.9.4
torch @ https://download.pytorch.org/whl/cu118/torch-2.0.1%2Bcu118-cp310-cp310-linux_x86_64.whl#sha256=a7a49d459bf4862f64f7bc1a68beccf8881c2fa9f3e0569608e16ba6f85ebf7b
````
- Python 3.10.12
