Anywhere where I can read more about the `device_map` kwarg in `from_pretrained`?
I’ve recently been trying to work with large models and discovered that usingdevice_map="auto"is an easy way to delegate loading large models onto GPU devices without me having to use.towith a lot of other configuration.I want to learn a bit more about how to design device maps but it seems like the documentation doesn’t have it (I tried following the hyperlink from“Handling Big Models for Inference”but it leads toa 404 page).I tried taking a look at the source code formodeling_utils.PretrainedModel.from_pretrainedbut am having some trouble getting a better grasp.Is there anywhere else that I could look to learn more about how device mapping works?
2024-01-05T07:44:08Z
huggingface.coHandling big models for inferenceWe’re on a journey to advance and democratize artificial intelligence through open source and open science.
2024-01-07T23:31:55Z
This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.