Create README.md
Browse files
README.md
ADDED
|
@@ -0,0 +1,15 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
base_model: mistralai/Mistral-Small-Instruct-2409
|
| 3 |
+
---
|
| 4 |
+
|
| 5 |
+
[EXL3](https://github.com/turboderp-org/exllamav3) quantization of [Mistral-Small-Instruct-2409](https://huggingface.co/mistralai/Mistral-Small-Instruct-2409), 4 bits per weight.
|
| 6 |
+
|
| 7 |
+
### HumanEval (argmax)
|
| 8 |
+
|
| 9 |
+
| Model | Q4 | Q6 | Q8 | FP16 |
|
| 10 |
+
| ---------------------------------------------------------------------------------------------------------------- | ---- | ---- | ---- | ---- |
|
| 11 |
+
| [Mistral-Small-Instruct-2409-exl3-3bpw](https://huggingface.co/isogen/Mistral-Small-Instruct-2409-exl3-3bpw) | 76.8 | 74.4 | 76.2 | 75.6 |
|
| 12 |
+
| [Mistral-Small-Instruct-2409-exl3-3.5bpw](https://huggingface.co/isogen/Mistral-Small-Instruct-2409-exl3-3.5bpw) | 73.8 | 75.6 | 75.0 | 75.6 |
|
| 13 |
+
| [Mistral-Small-Instruct-2409-exl3-4bpw](https://huggingface.co/isogen/Mistral-Small-Instruct-2409-exl3-4bpw) | 78.7 | 78.7 | 79.3 | 79.3 |
|
| 14 |
+
| [Mistral-Nemo-Instruct-2407-exl3-4bpw](https://huggingface.co/isogen/Mistral-Nemo-Instruct-2407-exl3-4bpw) | 74.4 | 72.6 | 73.2 | 72.0 |
|
| 15 |
+
| [Mistral-Nemo-Instruct-2407-exl3-6bpw](https://huggingface.co/isogen/Mistral-Nemo-Instruct-2407-exl3-6bpw) | 70.7 | 69.5 | 69.5 | 68.9 |
|