Quntisation LLM models
Am I correct in understanding that Q_4_K_S quantization means 4-bit quantization with k-quantization(K) and a small(S) block size? What does a small block size mean? Q_3 means 3 bit quantization? Thanks!
2024-11-18T08:51:49Z
I can’t say I understand it correctly either, so I tried searching for it. Well, if you’re in trouble, use Q4_K_M.https://www.reddit.com/r/LocalLLaMA/comments/1d1sc50/gguf_weight_encoding_suffixes_is_there_a_guide/gist.github.comhttps://gist.github.com/Artefact2/b5f810600771265fc1e39442288e8ec9README.md# Which GGUF is right for me? (Opinionated)

Good question! I am collecting human data on how quantization affects outputs. See here for more information: https://github.com/ggerganov/llama.cpp/discussions/5962

In the meantime, use the largest that fully fits in your GPU. If you can comfortably fit Q4_K_S, try using a model with more parameters.

# llama.cpp feature matrix

See the wiki upstream: https://github.com/ggerganov/llama.cpp/wiki/Feature-matrixThis file has been truncated.show originalMedium – 9 Oct 24Understanding the GGUF Format: A Comprehensive GuideIntroductionReading time: 15 min readmaartengrootendorst.comA Visual Guide to Quantization - Maarten GrootendorstMy Personal Website