qwen3-4b-gdn-hybrid-stage2b-kd

Stage-2b endpoint — off-policy forward-KL KD through 32k context; the starting checkpoint for all stage-3 arms. Strong teacher-forced fit (wikitext-2 PPL 12.24 vs teacher 12.09; 0-shot avg 0.632) but broken generation behavior: MATH-500 thinking 0.000 (100% non-termination), RULER vt@32k 0.00 — the evaluation-blind-spot exhibit.

Part of a study converting full-attention Qwen3-4B into a GDN (gated DeltaNet) hybrid (27 of 36 layers converted, uniform 1:4 retention) and recovering capability via staged distillation, culminating in on-policy distillation (OPD): student rollouts scored per-token by the frozen teacher under reverse KL. Checkpoints are stock Qwen3NextForCausalLM — load with AutoModelForCausalLM (transformers >= 4.57). vLLM serving needs a small registry shim (all-dense MoE + tied-embeddings fixes) from the project repo.

Pipeline: surgery/init -> stage-1 hidden-state alignment (100M tok) -> stage-2a forward-KL KD @4k (600M) -> stage-2b long-context KD @32k (294M) -> stage-3 OPD (200M gen tokens, horizon 512->16k, token-keyed LR). Teacher/reference: Qwen/Qwen3-4B (thinking mode).