Is it possible to push to HuggingFace on every checkpoint while training?
Hi all!I’m trying to use one of the example scripts under the transformers library. In particular, this one:github.comtransformers/examples/pytorch/image-classification at main ·...main/examples/pytorch/image-classification🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX. - transformers/examples/pytorch/image-classification at main · huggingface/transformersAt the moment, I’m using the MNIST dataset, just to keep things simple. I’ve been reading the documentation, but I can’t seem to figure out if it is possible. As per my title, I would like to push to HuggingFace on every checkpoint, and include any optimizer / scheduler internal data alongside the checkpoint, so I can pull from HuggingFace and pick up where I left off.Is this something that is doable right now? Or should I write my own custom script to get this functionality?Thanks in advance,Farley
2022-09-06T12:51:17Z
Just addpush_to_hub=Trueto yourTrainingArguments.
2022-09-10T19:23:48Z
@sguggerThank you for the reply! Actually, what I’ve found is thatpush_to_hub=Truelaunches a backgroundgit push. I think the problem is that sometimes thegit pushprocess hangs. Is there any way to debug thatgitprocess, to see why it does not finish pushing?Also, just for context, I can sometimes go into the directory and usegit pushwith success. But also, I’ve found that only the first checkpoint produces agit commit. While trying to debug the backgroundgit push, I noticed that sometimes there can be 3 or 4 checkpoints created, but no commits happening for them. So only the first checkpoint gets pushed.Let me know if there’s any extra info I can provide. I might try and copy one of therun_*.pyscripts and writing some custom Python code to solve this problem.