Update README.md
Browse files
README.md
CHANGED
@@ -16,7 +16,7 @@ base_model:
|
|
16 |
---
|
17 |
|
18 |
[Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B) quantized with [torchao](https://huggingface.co/docs/transformers/main/en/quantization/torchao) int4 weight only quantization, using [hqq](https://mobiusml.github.io/hqq_blog/) algorithm for improved accuracy, by PyTorch team.
|
19 |
-
Use it directly or serve using [vLLM](https://docs.vllm.ai/en/latest/) for 62% VRAM reduction and 1.2x speedup on A100 GPUs.
|
20 |
|
21 |
# Inference with vLLM
|
22 |
Install vllm nightly and torchao nightly to get some recent changes:
|
|
|
16 |
---
|
17 |
|
18 |
[Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B) quantized with [torchao](https://huggingface.co/docs/transformers/main/en/quantization/torchao) int4 weight only quantization, using [hqq](https://mobiusml.github.io/hqq_blog/) algorithm for improved accuracy, by PyTorch team.
|
19 |
+
Use it directly or serve using [vLLM](https://docs.vllm.ai/en/latest/) for 62% VRAM reduction (6.27 GB needed) and 1.2x speedup on A100 GPUs.
|
20 |
|
21 |
# Inference with vLLM
|
22 |
Install vllm nightly and torchao nightly to get some recent changes:
|