#
VPTQ

Try VPTQ on Hugging Face! Try VPTQ on Google Colab! Know more about VPTQ on ArXiv!

Vector Post-Training Quantization (VPTQ) is a novel Post-Training Quantization method that leverages Vector Quantization to high accuracy on LLMs at an extremely low bit-width (<2-bit). VPTQ can compress 70B, even the 405B model, to 1-2 bits without retraining and maintain high accuracy.

- Better Accuracy on 1-2 bits, (405B @ <2bit, 70B @ 2bit)
- Lightweight Quantization Algorithm: only cost ~17 hours to quantize 405B Llama-3.1
- Agile Quantization Inference: low decode overhead, best throughput, and TTFT

Inference support for VPTQ is released in the `vptq` library. Make sure to install it to run the models:

    pip install vptq

The library provides efficient kernels for NVIDIA/AMD GPU inference.

To run VPTQ models simply load a model that has been quantized with VPTQ:

##
Inference example

Run Llama 3.1 70b on RTX4090 (24G @ ~2bits) in real time

    from transformers import AutoTokenizer, AutoModelForCausalLM

    quantized_model = AutoModelForCausalLM.from_pretrained(
        "VPTQ-community/Meta-Llama-3.1-70B-Instruct-v16-k65536-65536-woft",
        torch_dtype="auto",
        device_map="auto"
    )
    tokenizer = AutoTokenizer.from_pretrained("VPTQ-community/Meta-Llama-3.1-70B-Instruct-v16-k65536-65536-woft")
    input_ids = tokenizer("hello, it's me", return_tensors="pt").to("cuda")
    out = model.generate(**input_ids, max_new_tokens=32, do_sample=False)

##
Quantize your own model

VPTQ algorithm early-released at VPTQ , and checkout the tutorial.

##
Early Results from Tech Report

VPTQ achieves better accuracy and higher throughput with lower quantization overhead across models of different sizes. The following experimental results are for reference only; VPTQ can achieve better outcomes under reasonable parameters, especially in terms of model accuracy and inference speed.

Model | bitwidth | W2↓ | C4↓ | AvgQA↑ | tok/s↑ | mem(GB) | cost/h↓
--- | --- | --- | --- | --- | --- | --- | ---
LLaMA-2 7B | 2.02 | 6.13 | 8.07 | 58.2 | 39.9 | 2.28 | 2
| 2.26 | 5.95 | 7.87 | 59.4 | 35.7 | 2.48 | 3.1
LLaMA-2 13B | 2.02 | 5.32 | 7.15 | 62.4 | 26.9 | 4.03 | 3.2
| 2.18 | 5.28 | 7.04 | 63.1 | 18.5 | 4.31 | 3.6
LLaMA-2 70B | 2.07 | 3.93 | 5.72 | 68.6 | 9.7 | 19.54 | 19
| 2.11 | 3.92 | 5.71 | 68.7 | 9.7 | 20.01 | 19

##
More Models in VPTQ-community

⚠️ The repository only provides a method of model quantization algorithm.

⚠️ The open-source community VPTQ-community provides models based on the technical report and quantization algorithm.

Quick Estimation of Model Bitwidth (Excluding Codebook Overhead):

- Model Naming Convention: The model’s name includes the vector length $v$, codebook (lookup table) size, and residual codebook size. For example, “Meta-Llama-3.1-70B-Instruct-v8-k65536-256-woft” is “Meta-Llama-3.1-70B-Instruct”, where:

  - Vector Length: 8
  - Number of Centroids: 65536 (2^16)
  - Number of Residual Centroids: 256 (2^8)

- Equivalent Bitwidth Calculation:

  - Index: log2(65536) = 16 / 8 = 2 bits
  - Residual Index: log2(256) = 8 / 8 = 1 bit
  - Total Bitwidth: 2 + 1 = 3 bits

- Model Size Estimation: 70B * 3 bits / 8 bits per Byte = 26.25 GB

- Note: This estimate does not include the size of the codebook (lookup table), other parameter overheads, and the padding overhead for storing indices. For the detailed calculation method, please refer to Tech Report Appendix C.2.

Model Series | Collections | (Estimated) Bit per weight
--- | --- | ---
Llama 3.1 Nemotron 70B Instruct HF | HF 🤗 | 4 bits 3 bits 2 bits (1) 2 bits (2) 1.875 bits 1.625 bits 1.5 bits
Llama 3.1 8B Instruct | HF 🤗 | 4 bits 3.5 bits 3 bits 2.3 bits
Llama 3.1 70B Instruct | HF 🤗 | 4 bits 3 bits 2.25 bits 2 bits (1) 2 bits (2) 1.93 bits 1.875 bits 1.75 bits
Llama 3.1 405B Instruct | HF 🤗 | 4 bits 3 bits 2 bits 1.875 bits 1.625 bits 1.5 bits (1) 1.5 bits (2) 1.43 bits 1.375 bits
Mistral Large Instruct 2407 (123B) | HF 🤗 | 4 bits 3 bits 2 bits (1) 2 bits (2) 1.875 bits 1.75 bits 1.625 bits 1.5 bits
Qwen 2.5 7B Instruct | HF 🤗 | 4 bits 3 bits 2 bits (1) 2 bits (2) 2 bits (3)
Qwen 2.5 14B Instruct | HF 🤗 | 4 bits 3 bits 2 bits (1) 2 bits (2) 2 bits (3)
Qwen 2.5 32B Instruct | HF 🤗 | 4 bits 3 bits 2 bits (1) 2 bits (2) 2 bits (3)
Qwen 2.5 72B Instruct | HF 🤗 | 4 bits 3 bits 2.38 bits 2.25 bits (1) 2.25 bits (2) 2 bits (1) 2 bits (2) 1.94 bits
Reproduced from the tech report | HF 🤗 | Results from the open source community for reference only, please use them responsibly.
Hessian and Inverse Hessian Matrix | HF 🤗 | Collected from RedPajama-Data-1T-Sample, following Quip#

< > Update on GitHub