update readme

README.md

@@ -0,0 +1,30 @@
+---
+base_model:
+- mistralai/Mistral-Large-Instruct-2407
+pipeline_tag: text-generation
+tags:
+- mistral
+- 3bit
+---
+This is a 3bit AutoRound GPTQ version of Mistral-Large-Instruct-2407.
+This conversion used model-*.safetensors.
+
+Quantization script (it takes around 520 GB RAM and A40 GPU 40GB around 20 hours to convert):
+```
+from transformers import AutoModelForCausalLM, AutoTokenizer
+import torch
+model_name = "mistralai/Mistral-Large-Instruct-2407"
+model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.float16)
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+
+from auto_round import AutoRound
+
+bits, group_size, sym = 3, 128, True
+
+autoround = AutoRound(model, tokenizer, nsamples=256, iters=512, low_gpu_mem_usage=True, batch_size=4, bits=bits, group_size=group_size, sym=sym,
+                     device='cuda')
+autoround.quantize()
+output_dir = "./Mistral-Large-Instruct-2407-3bit"
+autoround.save_quantized(output_dir, format='auto_gptq', inplace=True)
+
+```