Does the HF Trainer class support multi-node training?
Does the HFTrainerclass support multi-node training? Or only single-host multi-GPU training?
2021-04-15T16:10:19Z
It supports both single-node and multi-node distributed training with the PyTorch launcher (torch.distributed.launch)
2021-04-15T16:13:44Z
thanks! who is doing the cross-instance allreduce? PyTorch DDP? Or Horovod? or some custom HF allreduce? any sample?
2021-04-15T16:27:12Z
It’s standard PyTorch DDP behind the scenes.
2023-01-09T21:03:35Z
@sguggerI tried the distributed training per thisHF blogand it failed with multi-node training. It gave an mpirun error when i tried to use multiple instances (similar to when i tried theAWS blog). In both cases i tried with ml.p3.16xlarge and ml.p3dn.24xlarge instances.Edit: I tried as written and with all variations of current PyTorch/HF URI containers (can’t link due to new account, limited to 2)Any ideas on how to fix?image2136×538 69.3 KB