Clarifying multi-GPU memory usage
Am I reading this thread (Training using multiple GPUs) correctly? I interpret that to mean:Training a model with batch size 16 on one GPU is equivalent to running a model with batch size 4 on 4 GPUsIs that correct? And does it differ betweenDataParallelandDistributedDataParallelmodes?
2020-11-05T22:22:55Z
It is correct. The difference betweenDataParallelandDistributedDataParallelis (in your current example):inDataParallelmode you have to set the batch size to 16 for your data loaders.inDistributedDataParallelyou have to set the batch size to 4 for your data loaders.