Dataloader_num_workers in a torch.distributed setup using HF Trainer
Hi everyone - super quick question. I looked around and I couldn’t find this previously asked, but my apologies if I missed something!Wondering if I have HF trainer set up using torch.distributed.launch on 8 gpus… if my dataloader_num_workers = 10… is that 10 total processes for dataloaders or 10*8=80 processes?Thank you again!
2022-01-19T19:12:04Z
That’s correct.
2022-01-19T19:24:17Z
Sorry… which one… 10 or 80? (Is it num workers * n_gpu or num_workers?) thank you!!
2022-01-19T19:24:44Z
10 * 8 = 80
2022-01-19T19:45:25Z
thank you!!!