Very Basic Questions about Revisions
Hi all, I am very new to this space (programming in general) and so learning about models and pipelines is quite fascinating.I am also new to this ‘git’ concept which I find quite confusing. When I upload a model to HF it gets stored as a “commit” or “revision”. That is pretty cool, since I can go back to an earlier commit and retrieve older model files before it got ruined by overtraining.But I am then concerned about the storage I am taking up on HF with some of these older versions.How do I permanently delete old commits or revisions through CLI, GUI or python? So specifically if I have 5 commits to date on model, i may want to delete commits 3 and 4, but keep 1, 2 and 5.Hope someone can help.
2023-04-25T16:48:05Z
Note that git only stores the difference (diff) between files, not the files themselves. However, if you update an entire neural network (all the weights), then each commit will have its ownpytorch_model.binfile, which can be several gigabytes large.In git, you can squash commits together. So in your case, you could squash commits 3, 4 and 5 into a single commit, which will only store the latest version (commit 5). So to do that, first clone a repository from the hub to your local computer:git lfs install
GIT_LFS_SKIP_SMUDGE=1 git clone <URL>You can add the GIT_LFS_SKIP_SMUDGE=1 to not load the big files.Next, squash the last 3 commits into one as explainedhere:git reset --soft HEAD~3 && git committhen you can push the updates back to the HF hub:git push
2023-04-25T16:59:44Z
Hi and thank you for your reply!However, if you update an entire neural network (all the weights), then each commit will have its ownpytorch_model.binfile, which can be several gigabytes largeExactly! that was my concernIn git, you can squash commits together. So in your case, you could squash commits 3, 4 and 5 into a single commit, which will only store the latest version (commit 5).Understood. I appreciate the info and the related code/link! I will try it out on some dummy files before squashing the actual model folder.