Memory usage during training
Hi,

I would like to know if there is any documentation on memory usage during training. I have a base of approximately 1000 hours and there are times when memory usage reaches 160GB (which is my current limit). I'm afraid this will interfere with training, hence my doubt. I also notice that this increase is constant each time I grow my database and the amount of GPUs

NeMo Version: 1.3.0
GPUs: 5
CPU Cores: 64
RAM: 160 GB
