Instructions to use addansee2/gemma-3-4b-it-qat-q4_0-unquantized-heretic with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use addansee2/gemma-3-4b-it-qat-q4_0-unquantized-heretic with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="addansee2/gemma-3-4b-it-qat-q4_0-unquantized-heretic")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("addansee2/gemma-3-4b-it-qat-q4_0-unquantized-heretic")
model = AutoModelForMultimodalLM.from_pretrained("addansee2/gemma-3-4b-it-qat-q4_0-unquantized-heretic", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use addansee2/gemma-3-4b-it-qat-q4_0-unquantized-heretic with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "addansee2/gemma-3-4b-it-qat-q4_0-unquantized-heretic"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "addansee2/gemma-3-4b-it-qat-q4_0-unquantized-heretic",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/addansee2/gemma-3-4b-it-qat-q4_0-unquantized-heretic

SGLang

How to use addansee2/gemma-3-4b-it-qat-q4_0-unquantized-heretic with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "addansee2/gemma-3-4b-it-qat-q4_0-unquantized-heretic" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "addansee2/gemma-3-4b-it-qat-q4_0-unquantized-heretic",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "addansee2/gemma-3-4b-it-qat-q4_0-unquantized-heretic" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "addansee2/gemma-3-4b-it-qat-q4_0-unquantized-heretic",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use addansee2/gemma-3-4b-it-qat-q4_0-unquantized-heretic with Docker Model Runner:
```
docker model run hf.co/addansee2/gemma-3-4b-it-qat-q4_0-unquantized-heretic
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

This is a decensored version of google/gemma-3-4b-it-qat-q4_0-unquantized, made using Heretic v1.1.0

Abliteration parameters

Parameter	Value
direction_index	per layer
attn.o_proj.max_weight	1.27
attn.o_proj.max_weight_position	32.22
attn.o_proj.min_weight	1.04
attn.o_proj.min_weight_distance	18.63
mlp.down_proj.max_weight	1.29
mlp.down_proj.max_weight_position	20.41
mlp.down_proj.min_weight	0.92
mlp.down_proj.min_weight_distance	9.75

Performance

Metric	This model	Original model (google/gemma-3-4b-it-qat-q4_0-unquantized)
KL divergence	0.4659	0 (by definition)
Refusals	3/100	97/100

Gemma 3 model card

Model Page: Gemma

This repository corresponds to the 4B instruction-tuned version of the Gemma 3 model using Quantization Aware Training (QAT).

The checkpoint in this repository is unquantized, please make sure to quantize with Q4_0 with your favorite tool

Thanks to QAT, the model is able to preserve similar quality as bfloat16 while significantly reducing the memory requirements to load the model.

Downloads last month: 31

Safetensors

Model size

4B params

Tensor type

BF16

Model tree for addansee2/gemma-3-4b-it-qat-q4_0-unquantized-heretic

Base model

google/gemma-3-4b-pt

Finetuned

google/gemma-3-4b-it

Finetuned

google/gemma-3-4b-it-qat-q4_0-unquantized

Finetuned

(12)

this model

Quantizations

3 models