Using the SpinQuant quantization method from https://github.com/facebookresearch/SpinQuant, I quantized the Llama-3.2-3B-Instruct-abliterated model from https://huggingface.co/huihui-ai/Llama-3.2-3B-Instruct-abliterated.
This quantization is for on-device deployment to Android apps with Executorch.
To make it easier for everyone to quickly test and deploy the Executorch on-device demo, I've also converted the quantized PTH file to PTE format and uploaded it.
2025-02-16 19:40:24,099 - spinquant - INFO - wiki2 ppl is: 11.502239227294922
Inference Providers
NEW
This model is not currently available via any of the supported Inference Providers.
The model cannot be deployed to the HF Inference API:
The model has no library tag.
Model tree for JamePeng2023/Llama-3.2-3B-Instruct-abliterated-SpinQuant-w4a8
Base model
meta-llama/Llama-3.2-3B-Instruct