Device_map="auto"
I was wondering if this parameter is only set for inference as the document states (Handling big models for inference) or does it actually have an effect during training?Thanks!
2023-08-06T05:49:30Z
It loads a model onto multiple GPUs. Once loaded, the model can be run forward or backward. I have only used ”auto” for training as of yet, and it works.If you refer to this section:This only supports the inference of your model, not training. Most of the computation happens behindtorch.no_grad()context managers to avoid spending some GPU memory with intermediate activations.I think it only applies to the offloading to CPU or disk mechanism, but not when the full model can be loaded onto several GPUs.
2023-08-07T14:51:30Z
Thanks@hansekbrandthat is helpful.With more search, I think it became clear to me that device_map=“auto” is doing naive MP for training (ref:Make all Transformer models compatible with model parallelism · Issue #22561 · huggingface/transformers · GitHub).
2023-08-07T14:53:33Z
Kenkentron:is doing naive MP for trainingCorrect, any form of distributed training aside MP is not supported, and as of the next version will raise a proper error if you try to do so:github.com/huggingface/accelerateExpand device-map warninghuggingface:main←huggingface:device-mapopened01:06PM - 07 Aug 23 UTCmuellerzr+22-9Since we're happy with the warning, expands it to include any call to `prepare`…(and `prepare_model`) and refactors it to it's own function.

Will ensure that https://github.com/huggingface/accelerate/issues/1815 will raise the right err
2023-08-09T17:04:09Z
Thanks@muellerzr, could you also take a look at a related problem I haveZeRO uses more RAM than DDP??
2024-09-25T09:27:22Z
training mode does not support device_map=“auto”, it will throw an error suggesting you load model on single device.