Cuda memory imbalanced in multiple GPUs, and multiple process in device 0
Hello, I am using v100 * 8 to finetune the deberta-v3-large model for sequence classification, I use transformers and Trainer API to train after usingaccelerate configto specify a deepspeed config, I useaccelerate launch main.pyto start training, but the process was not initialized properly, and the cuda memory is not balanced at all.e75869f0f38e30aa41bedbe78cbb8b681080×1920 411 KBHere is my config fileIMG_0195.HEIC3024×4032 2.88 MBIt is pretty weird, I just use a huggingface trainer API to code