jerryzh168 commited on
Commit
92abbc1
·
verified ·
1 Parent(s): 1bd5c07

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +1 -1
README.md CHANGED
@@ -16,7 +16,7 @@ base_model:
16
  ---
17
 
18
  [Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B) quantized with [torchao](https://huggingface.co/docs/transformers/main/en/quantization/torchao) int4 weight only quantization, using [hqq](https://mobiusml.github.io/hqq_blog/) algorithm for improved accuracy, by PyTorch team.
19
- Use it directly or serve using [vLLM](https://docs.vllm.ai/en/latest/) for 62% VRAM reduction and 1.2x speedup on A100 GPUs.
20
 
21
  # Inference with vLLM
22
  Install vllm nightly and torchao nightly to get some recent changes:
 
16
  ---
17
 
18
  [Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B) quantized with [torchao](https://huggingface.co/docs/transformers/main/en/quantization/torchao) int4 weight only quantization, using [hqq](https://mobiusml.github.io/hqq_blog/) algorithm for improved accuracy, by PyTorch team.
19
+ Use it directly or serve using [vLLM](https://docs.vllm.ai/en/latest/) for 62% VRAM reduction (6.27 GB needed) and 1.2x speedup on A100 GPUs.
20
 
21
  # Inference with vLLM
22
  Install vllm nightly and torchao nightly to get some recent changes: