Is it normal of more memory use of DistributedDataParallel than single
Hello , I am new here.I try to Fine-turn MBart model ( mbart-large-cc25 ), My device : one pc(ubuntu), gpu(10GB) memory * 2When i fine-turn on single gpu, first load model to gpu , only cost 3(GB) memory and start train it, increase to 8(GB), so i can fine-turn with small batch.I use DistributedDataParallel to have more batch in fine-turnBut, i load model to 2 gpu , than cost 7GB both, didn’t start training.Is it normal ?Here is my init memory codeclass SummarizationModule() :
    def __init__(self,local_rank) -> None:
        self.mBartModel = MBartForConditionalGeneration.from_pretrained("facebook/mbart-large-cc25")
        self.mBartTokenizer = MBartTokenizer.from_pretrained("facebook/mbart-large-cc25", src_lang="en_XX", tgt_lang="zh_CN")
        self.mLocal_rank = local_rank
        torch.cuda.set_device(self.mLocal_rank)
        print('Rank : [' , args.local_rank , '] is ready.')
        os.environ['CUDA_VISIBLE_DEVICES'] = '0,1'
        os.environ['MASTER_ADDR'] = 'localhost'
        os.environ['MASTER_PORT'] = '8787'
        torch.distributed.init_process_group(backend="nccl", init_method="env://", world_size=1 ,rank=self.mLocal_rank )
        self.mBartModel = self.mBartModel.cuda()
        self.mBartModel = DistributedDataParallel(self.mBartModel, find_unused_parameters=True)
        os.system('nvidia-smi')

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument('--local_rank', default=-1, type=int, help='node rank for distributed training')
    args = parser.parse_args()
    mModel = SummarizationModule(args.local_rank)Use :CUDA_VISIBLE_DEVICES=0,1 python3 -m torch.distributed.launch --nproc_per_node=2 main_mbart_multi_gpu.py
2021-06-21T13:09:24Z
There is a slight overhead when usingDistributedDataParallelso it’s normal to see a bit more GPU usage yes.
2021-06-22T01:04:51Z
Thank you for your help.