|
|
--- |
|
|
base_model: TheDrummer/Anubis-70B-v1.1 |
|
|
base_model_relation: quantized |
|
|
quantized_by: ArtusDev |
|
|
license: llama3.3 |
|
|
pipeline_tag: text-generation |
|
|
library_name: transformers |
|
|
tags: |
|
|
- fp8 |
|
|
- fp8-dynamic |
|
|
--- |
|
|
|
|
|
## FP8 Quant of TheDrummer/Anubis-70B-v1.1 |
|
|
|
|
|
FP8 quant of [TheDrummer/Anubis-70B-v1.1](https://huggingface.co/TheDrummer/Anubis-70B-v1.1) using <a href="https://github.com/vllm-project/llm-compressor/">llm-compressor</a> for quantization. |
|
|
|
|
|
### Downloading quants with huggingface-cli |
|
|
|
|
|
<details> |
|
|
<summary>Click to view download instructions</summary> |
|
|
|
|
|
Install hugginface-cli: |
|
|
|
|
|
```bash |
|
|
pip install -U "huggingface_hub[cli]" |
|
|
``` |
|
|
|
|
|
Download quant by targeting the specific quant revision (branch): |
|
|
|
|
|
``` |
|
|
huggingface-cli download ArtusDev/TheDrummer_Anubis-70B-v1.1-FP8-Dynamic --local-dir ./ |
|
|
``` |
|
|
</details> |
|
|
|
