.bin to safetensors without publishing it on hub?
I have followedTrain a diffusion modelto create own model just for learning. In the end I got “.bin” file, but I cannot load it on Stable Diffusion.I found out that there is a converter inTrain a diffusion modelbut it asks me model_id on hub. Is there any way to convert this “.bin” to .safetensors locally without any kind of uploadings? If so, with what tools?
2023-05-15T15:02:16Z
Hi! You can check thesourcecode of theConvert to Safetensorsspace to find how to do this locally.
2023-05-15T15:22:19Z
Thank you very much, this did it! I cloned the source and modified convert.py to suit my needs.In case somebody else wants to make this work also then I changed the code to from main function to call convert_file(“diffusion_pytorch_model.bin”, “output.safetensors”) and commented out#os.makedirs(dirname, exist_ok=True) line.I also removed lots of functions before I noticed that I could just call the convert_file directly.
2024-08-26T16:58:54Z
I wonder does it works on llm models eg:Llama3.18B,I got a model with four.bin files
2024-08-27T04:09:53Z
I think it will work if you change the code slightly. (Because the filename is a magic word.)Or if you use huggingface_hub.save_torch_model/ save_torch_state_dict instead of safetensors.torch.save_file, you’ll be able to handle sharded files seamlessly.huggingface.coSerializationWe’re on a journey to advance and democratize artificial intelligence through open source and open science.But the load function isnot yet implemented, so you’ll have to make your own it…I see the author of the space above is also doing it manually.If not safetensors, torch.save and torch.load should be fine.with torch.no_grad():
    def load_sharded_safetensors(path: str):
        import glob
        sd = {}
        try:
            for filepath in glob.glob(f"{path}/*.safetensors"):
                sharded_sd = load_file(str(filepath), device="cpu")
                for k, v in sharded_sd.items():
                    sharded_sd[k] = v.to(device="cpu")
                sd = sd | sharded_sd.copy()
                del sharded_sd
        except Exception as e:
                print(e)
        return sdIf the model is too large to fit in RAM, it is better to load, convert, save, and repeat for each file.
2024-12-18T22:07:09Z
can also be used this wayClone source -  git cloneConvert to Safetensors - a Hugging Face Space by safetensorsadd, Create new python file on repo, The name could be like this convert_bin_to_safetensor.pyadd codes at below#!/usr/bin/env python
# -*- coding: utf-8 -*-
from convert import convert_file

if __name__ == "__main__":
    convert_file("/path/to/unet/diffusion_pytorch_model.bin",
    "/path/to/unet/diffusion_pytorch_model.safetensors",
    discard_names=[])And run on terminalpython3 convert_bin_to_safetensor.py
2024-12-19T01:35:38Z
The load function of the sharded model of torch has started to be implemented in HF Hub.huggingface.co@celinah on Hugging Face: "🚀 We've just dropped a new release v0.27.0 of the…"We’re on a journey to advance and democratize artificial intelligence through open source and open science.huggingface.coSerializationWe’re on a journey to advance and democratize artificial intelligence through open source and open science.