why consolidated.safetensors is required?
On Huggingface site, most models with safetensors format don't need a model's consolidated.safetensors and vllm works with them perfectly. But voxtral model doesn't follow the same way. We have our firm's policy that limits the file size for downloading. I could not download the consolidated.safetensors of mistralai/Voxtral-Small-24B-2507.
Any ideas?
I tend to use snapshot_download to avoid the consolidated files (Mistral), pytorch binaries (llama) and left over training files (random models).
This will work for Voxtral-Small:
original_model="mistralai/Voxtral-Small-24B-2507"
original_model_path="Voxtral-Small-24B-2507"
from huggingface_hub import snapshot_download
model_path = snapshot_download(
repo_id=original_model,
local_dir=original_model_path,
allow_patterns=[
"*.json",
"*.safetensors",
"merges.txt",
"*.md",
"tokenizer.*"
],
ignore_patterns=[
"optimizer.pt",
"consolidated.safetensors", #<- This excludes the consolidated file.
"pytorch_model.bin",
"training_args.bin",
"scheduler.pt",
]
)
At least it's not Mistral-Large or Mixtral-8x22b lol
Thanks for the script you provided. I learned the idea although it didn't work in my restricted environment.
@Hansen-Wu
In that case, I guess re-upload it without the junk like I've done here: Gapeleon/Voxtral-Small-24B-2507
You're welcome to use my repo, but sounds like your firm wouldn't want you deploying weights from random users on huggingface.
Public repos are free so you could just upload it to your account (you could do the download -> upload on a colab instance for example.)