Instructions to use scottgl/Qwen3.5-122B-A10B-NVFP4-GB10 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use scottgl/Qwen3.5-122B-A10B-NVFP4-GB10 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="scottgl/Qwen3.5-122B-A10B-NVFP4-GB10")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("scottgl/Qwen3.5-122B-A10B-NVFP4-GB10")
model = AutoModelForMultimodalLM.from_pretrained("scottgl/Qwen3.5-122B-A10B-NVFP4-GB10", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use scottgl/Qwen3.5-122B-A10B-NVFP4-GB10 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "scottgl/Qwen3.5-122B-A10B-NVFP4-GB10"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "scottgl/Qwen3.5-122B-A10B-NVFP4-GB10",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/scottgl/Qwen3.5-122B-A10B-NVFP4-GB10

SGLang

How to use scottgl/Qwen3.5-122B-A10B-NVFP4-GB10 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "scottgl/Qwen3.5-122B-A10B-NVFP4-GB10" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "scottgl/Qwen3.5-122B-A10B-NVFP4-GB10",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "scottgl/Qwen3.5-122B-A10B-NVFP4-GB10" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "scottgl/Qwen3.5-122B-A10B-NVFP4-GB10",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use scottgl/Qwen3.5-122B-A10B-NVFP4-GB10 with Docker Model Runner:
```
docker model run hf.co/scottgl/Qwen3.5-122B-A10B-NVFP4-GB10
```

Qwen3.5-122B-A10B-NVFP4-GB10

Qwen3.5-122B-A10B optimized for NVIDIA DGX Spark (GB10, SM12.1) — all quantizations baked in.

This checkpoint is derived from Sehyo/Qwen3.5-122B-A10B-NVFP4 with additional post-quantizations applied and saved permanently into the weights. The result loads and serves immediately on GB10 without any runtime quantization overhead.

Why this exists

The base NVFP4 checkpoint leaves several layer types in BF16 to keep it hardware-agnostic. On GB10 (NVIDIA Grace Blackwell, SM12.1), those BF16 layers are a significant bottleneck:

GDN (GatedDeltaNet) layers are Qwen3.5's linear attention mechanism. They run on every token and constitute ~33% of decode compute time at BF16. Quantizing them to FP4/FP8 roughly halves their memory bandwidth cost with minimal accuracy impact.
The lm_head projects hidden states to a 248,320-token vocabulary at every decode step. At BF16 this reads ~1.5 GB of weights per token generated. Quantizing to FP8 halves this to ~750 MB/token — a direct throughput gain.

These optimizations are implemented in scottgl9/sglang-spark-gb10-optimizations as runtime post-quantization hooks. This checkpoint bakes them in so:

Startup is faster (no quantization at load time)
The exact quantized weights are reproducible and bit-identical every run
Other GB10 users can use this checkpoint without needing the custom SGLang fork

What's quantized and why

Layer	Original	Baked format	Why
MoE expert weights (gate/up/down)	NVFP4 (W4A4)	NVFP4 (W4A4)	From source checkpoint — preserved as-is
GDN `in_proj_qkv`, `in_proj_z`	BF16	NVFP4 (Marlin FP4)	Large projections; SM12.1 has native Marlin FP4 kernels. CUTLASS FP4 is broken on GB10.
GDN `in_proj_a`, `in_proj_b`, `out_proj`	BF16	FP8 (per-tensor, dynamic activations)	Smaller GDN projections; FP8 CUTLASS works correctly on SM12.1. cos_sim > 0.999, SNR = 31.5 dB vs BF16.
lm_head (vocabulary projection)	BF16	FP8 (dynamic activations)	3072 x 248,320 matmul at every decode step. FP8 halves DRAM reads vs BF16. Activations remain dynamic per-token for accuracy.
Attention Q/K/V/O	BF16	BF16	Standard attention; not the bandwidth bottleneck at batch=1.

Performance

~46 tok/s decode on a single NVIDIA DGX Spark (GB10, 128 GB unified memory) using SGLang with NEXTN speculative decoding (steps=2, draft_tokens=2).

Baseline without any of these optimizations: ~28 tok/s. The baked quantizations account for approximately +18 tok/s of that gain.

Hardware

Tested on: NVIDIA DGX Spark (ASUS Ascent GX10), GB10 Grace Blackwell, SM12.1, 128 GB unified memory.

Requirements:

SM12.1 (GB10) — Marlin FP4 kernels are SM12.1-specific
scottgl9/sglang-spark-gb10-optimizations SGLang fork
CUDA 13.x

Will NOT work on:

Standard H100/A100 — Marlin FP4 kernels require Blackwell SM12.1
Apple Silicon — Metal only, no CUDA path

Usage

# Load directly — no post-quant overhead at startup
SGLANG_QUANTIZE_LM_HEAD_FP8=0 ./sglang.sh Qwen3.5-NVFP4 \
    --model-path scottgl9/Qwen3.5-122B-A10B-NVFP4-GB10

SGLANG_QUANTIZE_LM_HEAD_FP8=0 tells SGLang to skip the lm_head FP8 post-quant (already baked in). All other sglang.sh flags remain unchanged.

With NEXTN speculative decoding (recommended):

SGLANG_QUANTIZE_LM_HEAD_FP8=0 ./sglang.sh Qwen3.5-NVFP4 \
    --model-path scottgl9/Qwen3.5-122B-A10B-NVFP4-GB10 \
    --speculative-algorithm NEXTN \
    --speculative-num-steps 2 \
    --speculative-num-draft-tokens 2

Source models

Base quantization: Sehyo/Qwen3.5-122B-A10B-NVFP4
Original model: Qwen/Qwen3.5-122B-A10B
SGLang GB10 fork: scottgl9/sglang-spark-gb10-optimizations

License

Apache 2.0 (same as Qwen3.5)

Downloads last month: 533

Safetensors

Model size

27B params

Tensor type

F32

I32

BF16

F8_E4M3

Model tree for scottgl/Qwen3.5-122B-A10B-NVFP4-GB10

Base model

Qwen/Qwen3.5-122B-A10B

Quantized

(152)

this model