Batchsize decreasing with each new GPU
Hi,

With every new GPU I add I find I'm forced to lower batchsize because of memory overflows. This is normal? 
For example with 1 GPU I can use 68 BatchSize, with 2 GPUs I can use 67 and with 5 I can use 62.

OS:
Ubuntu 18.04
NeMo 1.3.0
Python 3.8