How to quickly determine memory requirements for model
Hey all - as I browse models for those that suit my project, I am trying to quickly determine the memory requirements for running each model locally.  This seems like something many users would want to do and so there should be an obvious place to look on a model card to make this determination, but I don’t see any such place.  I don’t even see a place where the parameter count is consistently stated.  How should I approach this?
2023-06-19T16:23:40Z
Hi! You can find this info by checking the size ofpytorch_model.bin(ortf_model.h5/flax_model.msgpackfor TF/Flax models). These files can be sharded sometimes (ifpytorch_model.bin.index.jsonis present), in which case you need to sum up all the shards listed in the index file.PS:  For the parameter count to be displayed, the weights must be saved in thesafetensorsformat.
2023-06-24T09:40:32Z
Thanks very much for the help.
2023-09-21T18:07:10Z
How would I determine these requirements when I’m finetuning a model?  I’m finetuning a 780 MB model with a 5 MB dataset.  On my local machine this runs fine (16 GB ram) but when I use a GPU (Tesla T4 with 16 GB of GPU ram), I immediately get an OOM error when I launchtrainer.train().
2023-09-24T11:17:56Z
Have you tried Model Memory Calculator?Model Memory Utility - a Hugging Face Space by hf-accelerate
2023-09-24T12:57:43Z
Nice link, thanks.So finetuning should consume less memory than training, and training the model I’m using is just 4.57 GB according to the calculator.  So I’m probably misunderstanding something about the model architecture since I have 16 GB GPU RAM.image3276×965 127 KB
2023-11-07T05:44:55Z
pvelosipednikov:hould consume less memory than training, and training the model I’m using is just 4.57 GB according to tTotal memory required during training/tuning should depend on the batch size right? Why batch size is not an input to this utility?
2024-08-28T19:27:29Z
The batch size is automatically set to one as said in, “When training on a batch size of 1”.