Jon-Nielsen/DeepSeek-V4-Flash-0731-MXFP4-INT8

DeepSeek-V4-Flash-0731 MXFP4-INT8

Ampere (RTX 3090, sm86) has no native FP8 or MXFP4 tensor cores. The released 0731 checkpoint (FP8+MXFP4) using mainline vLLM has no officially supported loading path.

Using AppMana's vLLM fork, this quant and the included patches enables serving.

Tested on 8x RTX 3090 (24GB, sm86). See "What to expect" below.

What this is

DeepSeek-V4-Flash-0731 converted to a format that runs on Ampere GPUs:

• Experts: MXFP4 (E2M1) — lossless. On sm86, E2M1 is reinterpreted as signed INT4 and run through Ampere INT4 tensor cores via Marlin. E8M0 power-of-2 scales applied in epilogue. No quality loss vs original.
• Dense linears: FP8 E4M3 → INT8 (channelwise). Lossy but minimal: measured 40 dB signal-to-noise ratio (~1% mean relative error) on the original checkpoint's FP8 weights after round-trip. For comparison, a typical W4A16 quant measures ~20-30 dB.
• KV cache: int8_ds_mla. Doubles context vs fp8 (437K vs ~220K on 8x24GB).
• DSpark: 3 MTP draft stages preserved from 0731 source. Observed 0% draft acceptance with int8 KV on sm86. Disable for production.

How it was created

Converted with tools/ampere/dsv4_requant_checkpoint.py from the AppMana vllm-consumer-nvidia-platforms fork:

python tools/ampere/dsv4_requant_checkpoint.py \
  --src deepseek-ai/DeepSeek-V4-Flash-0731 \
  --dst ./output \
  --expert-format mxfp4 \
  --dense-int8-strategy channel \
  --device cuda:0 \
  --overwrite

MXFP4 path is 95% passthrough — expert weights copied byte-for-byte, only ~8 FP8 linears per shard requanted to INT8. Conversion takes ~15-30 min on a single GPU. See serving/convert.sh.

Source: deepseek-ai/DeepSeek-V4-Flash-0731 Fork: AppMana/forks-vllm-consumer-nvidia-platforms branch appmana/vllm-consumer-nvidia-platforms Similar quants: appmana/deepseek-v4-mxfp4-int8 (pre-0731, no DSpark), appmana/deepseek-v4-int4-int8 (0731, INT4 experts, faster but less headroom)

What to expect

Tested on 8x RTX 3090 (24GB, sm86), CUDA 13.1, torch 2.13.0+cu130:

Config	Context	Speed	Notes
This quant, int8 KV, FULL_DECODE_ONLY graphs	437K	62 tok/s	DSpark OFF
This quant, DSpark ON	62K	35 tok/s	0% DSpark acceptance

• Correct math, coherent long-form generation, working tool calls
• No quality evals yet — evaluate before production use
• GMU=0.93 is the sweet spot (0.95 always OOMs on 24GB cards)
• --max-num-batched-tokens 128 gives 100% Triton JIT cache hit

Verified on 8x RTX 3090 (sm86), PCIe connected @ 200W power limit, CUDA 13.1, torch 2.13.0+cu130:

• Single-stream decode: 62 tok/s, TTFT 0.19s
• 4x concurrent: 157 tok/s aggregate
• Context auto-fit: 436,992 tokens
• Functional: basic chat, reasoning (enable_thinking via chat_template_kwargs), tool calls, streaming, multi-turn memory, code generation, 16K-token long context — all pass
• GPU memory: 23.8/24.6 GB per card at GMU=0.93

No attempt was made to make DSpark work. We prioritized serving context.

No claims are made for suitability for any purpose.

The only intent is to provide inspiration for owners of 8x3090 rigs.

How to serve

Requires the AppMana fork with patches applied (see serving/patches/) and flash-mla==2.0.0+8ec3de6 installed.

The flash-mla wheel is served from https://appmana.github.io/forks-flash-mla-int/, which redirects to GitHub releases. If both are unavailable, the sm86 attention path has no fallback. Consider mirroring the wheel if you depend on this setup.

# Clone the fork and checkout the correct branch
git clone --branch appmana/vllm-consumer-nvidia-platforms \
  https://github.com/AppMana/forks-vllm-consumer-nvidia-platforms.git
cd forks-vllm-consumer-nvidia-platforms
# Pin to the commit this quant was tested against:
git checkout bd2bf884e4f960897e0cb9db54ff80072ef3fcfa

# Install torch (must match the fork's pin: 2.13.0)
pip install torch==2.13.0+cu130 \
  --extra-index-url https://download.pytorch.org/whl/cu130

# Install flash-mla (sm86 attention path hard-imports it, no fallback)
pip install flash-mla==2.0.0+8ec3de6 \
  --extra-index-url https://appmana.github.io/forks-flash-mla-int/

# Apply patches to the fork
git apply serving/patches/0001-expert-weight-loader.patch
git apply serving/patches/0002-conversion-tool-config.patch
# Patch 0003 is optional — the checkpoint's config.json already routes
# prefill through the Triton path, bypassing the crashy flash_mla int8
# prefill kernel. Only apply 0003 if you manually override the config
# to use the flash_mla prefill symbol.
# git apply serving/patches/0003-int8-prefill-safety.patch

# Build and install the fork (sm86 only — saves 3-5x build time vs all archs)
TORCH_CUDA_ARCH_LIST="8.6" pip install -e . --no-build-isolation

# Download the quant to the standard HF cache
hf download Jon-Nielsen/DeepSeek-V4-Flash-0731-MXFP4-INT8

# Serve (TP defaults to 8 — set to match your GPU count)
# MODEL can be a local path or the HF cache snapshot path.
VENV=.venv MODEL=./dsv4-mxfp4-int8-0731 \
  TP=8 GPU_MEM_UTIL=0.93 ENABLE_DSPARK=0 \
  bash serving/serve.sh

See serving/serve.sh for all configurable env vars.

Patches

Three patches to the AppMana fork. Only patch 1 is required to load the checkpoint; the checkpoint's config.json already includes the fixes from patches 2-3. Patch 2 is included for reproducibility (re-running conversion). Patch 3 is a safety net — only needed if you manually override the config to use the flash_mla int8 prefill symbol instead of the Triton path.

Patch 1: Pre-fused expert weight loader (required)

vllm/models/deepseek_v4/nvidia/model.py

MXFP4 checkpoints store pre-fused routed_experts.w13_weight as [E, 2*I, H/2] (gate+up fused). The fork's weight loader expected separate w1/w3 tensors. Patch adds a fallback that splits w13 and loads each half via the per-expert weight_loader. Without this, expert weights fail to load.

Patch 2: Conversion tool kernel block (reproducibility)

tools/ampere/dsv4_requant_checkpoint.py

The _write_config function always wrote the INT4 kernel block (6 symbols including marlin_act_int8_process_scales, which requires INT4 expert groups). MXFP4 checkpoints have FP4 experts — that block fails validation. Patch makes the kernel block conditional on expert_format:

• mxfp4: 2-symbol block (decode_int8 + Triton prefill), int8_ds_mla cache
• int4: full 6-symbol block (unchanged)

Patch 3: int8 prefill safety wrapper (safety net)

vllm/models/deepseek_v4/nvidia_imma/attention.py

The flash_mla int8 prefill C++ kernel (fwd_sparse_int8_prefill_mla in flash_mla_cuda.abi3.so) crashes on large prefill batches: aten::new_empty stable-ABI dispatcher error at ops.h:933. Small prompts work; large prompts crash. This was observed with flash-mla==2.0.0+8ec3de6; a future build may fix it. The int8 decode and fp8 prefill kernels are unaffected.

This checkpoint's config.json routes prefill through the Triton path (sparse_attention_triton) instead of the flash_mla int8 kernel, avoiding the crash entirely. The patch adds a @torch.compiler.disable wrapper around the flash_mla int8 prefill call — only needed if you manually configure the flash_mla prefill symbol instead of using the Triton path.

Serving directory

serving/
  serve.sh                              — generalized serve script
  convert.sh                            — conversion command
  patches/
    0001-expert-weight-loader.patch     — required to load checkpoint
    0002-conversion-tool-config.patch   — for re-running conversion
    0003-int8-prefill-safety.patch      — safety net for flash_mla prefill

---

Below is the original README from deepseek-ai/DeepSeek-V4-Flash-0731, included for reference only. The encoding/ and inference/ folders it mentions are not part of this repository.

Original DeepSeek-V4-Flash-0731 README

---

Technical Report👁️

Introduction

DeepSeek-V4-Flash-0731 is the official release of DeepSeek-V4-Flash, superseding the preview version, with substantially enhanced agentic capabilities. It has the same model structure as DeepSeek-V4-Flash-DSpark, i.e. it comes with a speculative decoding module attached.

DeepSeek-V4-Flash-0731 outperforms DeepSeek-V4-Pro (Preview) on benchmarks listed below despite its far smaller activated parameter count, and is broadly competitive with the strongest proprietary models available.

Benchmark	DeepSeek-V4-Flash-0731	DeepSeek-V4-Flash (Preview)	DeepSeek-V4-Pro (Preview)	GLM-5.2	Opus-4.8
Terminal Bench 2.1	82.7	61.8	72.1	81.0	85.0
NL2Repo	54.2	39.4	38.5	48.9	69.7
Cybergym	76.7	38.7	52.7	-	83.1
DeepSWE	54.4	7.3	12.8	46.2	58.0
Toolathlon-Verified	70.3	49.7	55.9	59.9	76.2
Agents' Last Exam	25.2	15.8	16.5	23.8	25.7
AutomationBench Public	25.1	10.8	12.8	12.9	27.2
DSBench-FullStack †	68.7	37.0	41.8	61.8	71.6
DSBench-Hard †	59.6	25.8	31.1	54.5	71.7

Notes:

1. For the Code Agent tasks among the public benchmarks above, DeepSeek-V4-Flash-0731 is evaluated with the minimal mode of DeepSeek Harness (to be released) as the agent framework, using the max reasoning effort level with temperature = 1.0, top_p = 0.95.
2. † DSBench-FullStack is an internal full-stack development test set; DSBench-Hard is an internal test set of difficult coding-agent problems.

Chat Template

This release does not include a Jinja-format chat template. Instead, we provide a dedicated encoding folder with Python scripts and test cases demonstrating how to encode messages in OpenAI-compatible format into input strings for the model, and how to parse the model's text output. Please refer to the encoding folder for full documentation.

The reasoning_effort parameter now supports three levels — low, high, and max — which control how much deliberation the model spends before answering.

A brief example:

from encoding_dsv4 import encode_messages, parse_message_from_completion_text

messages = [
    {"role": "user", "content": "hello"},
    {"role": "assistant", "content": "Hello! I am DeepSeek.", "reasoning_content": "thinking..."},
    {"role": "user", "content": "1+1=?"}
]

# messages -> string
prompt = encode_messages(messages, thinking_mode="thinking", reasoning_effort="max")

# string -> tokens
import transformers
tokenizer = transformers.AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-V4-Flash-0731")
tokens = tokenizer.encode(prompt)

How to Run with vLLM

DSpark speculative decoding is enabled with a single flag — add --speculative-config with method: dspark to your vLLM launch command:

--speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"greedy"}'

For example, the command below serves the model with vLLM on a single 4×GB300 node. See the vLLM recipe for detailed instructions and other hardware configurations.

vllm serve deepseek-ai/DeepSeek-V4-Flash-0731 \
  --trust-remote-code --kv-cache-dtype fp8 --block-size 256 \
  --data-parallel-size 4 --enable-expert-parallel \
  --moe-backend deep_gemm_mega_moe \
  --attention-config '{"use_fp4_indexer_cache": true}' \
  --speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"greedy"}'

How to Run with SGLang

Enable DSpark with --speculative-algorithm DSPARK and do not set a separate --speculative-draft-model-path as the target and draft weights therefore come from the same checkpoint. See the SGLang cookbook for detailed instructions, benchmarks and other hardwares configurations.

sglang serve \
  --trust-remote-code \
  --model-path deepseek-ai/DeepSeek-V4-Flash-0731 \
  --tp 4 \
  --moe-runner-backend flashinfer_mxfp4 \
  --speculative-algorithm DSPARK \
  --mem-fraction-static 0.90 \
  --chunked-prefill-size 4096 \
  --swa-full-tokens-ratio 0.1 \

How to Run Locally

Please refer to the inference folder for detailed instructions on running DeepSeek-V4 locally, including model weight conversion and interactive chat demos.

For local deployment, we recommend setting the sampling parameters to temperature = 1.0, with top_p = 0.95 for agentic scenarios and top_p = 1.0 otherwise. For the high and max reasoning effort levels, we recommend a maximum output length of 384K tokens.

License

This repository and the model weights are licensed under the MIT License.

Citation

@misc{deepseekai2026deepseekv4,
      title={DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence},
      author={DeepSeek-AI},
      year={2026},
}

Contact

If you have any questions, please raise an issue or contact us at service@deepseek.com.