license: mit
tags:
- comfyui
- minimax-h3
- text-to-video
- qwen3-vl
- text-encoder
- multilingual
base_model:
- Comfy-Org/MiniMax-H3
- Qwen/Qwen3-VL-4B-Instruct
- Qwen/Qwen3-VL-8B-Instruct
library_name: comfyui
ClipProj v3.1 — measured against the 32B's own variance
Four projection matrices that let a Qwen3-VL-4B or 8B replace the Qwen3-VL-32B text encoder of MiniMax H3.
15.0 GB → 4.6 GB, with no change to the diffusion model, the VAEs or the sampler.
This release is not about a new architecture. It is about finally knowing how good these things are, because the previous numbers could not tell me. This card is mostly the measurement, and the measurement changed three of my own conclusions.
Requires the custom node: github.com/nicolab28/ComfyUI-ClipProj
Read this before the tables: these are metrics, not verdicts
I do not speak these eleven languages. I cannot tell you whether a render sounds right, and I have not asked anyone who can. No native speaker has listened to any of the 297 speech renders on this page.
So nothing below is a judgement of quality. Every figure is a distance between two automatic transcriptions — what one machine wrote down from the reference, against what it wrote down from the projection. That is all it is, and it is worth being explicit about what that does and does not capture:
What the numbers do capture. Whether the same words and the same sounds come out. Two instruments are used precisely because each is wrong in a known direction: Whisper has a language model inside and corrects a slurred word into the most probable real one, so it under-reports pronunciation defects — a lower bound. ZIPA has no lexical decoder at all and counts every shift in realisation as an error, so it over-reports — an upper bound. What a listener would notice lies between them, and neither number alone is the answer.
What they do not capture. Prosody, rhythm, timbre, naturalness — everything that makes speech sound native rather than merely correct. A render scoring 98.8 here could still sound foreign to someone who speaks the language. These metrics cannot see that, and neither can I.
So read a score as "close to the 32B, according to this instrument" — never as "good". If you speak one of these languages, your ear outranks every table below, and I would genuinely like to hear what it tells you.
What changed in v3.1: giving every script its share
v3 was calibrated on a corpus that was overwhelmingly English, with the other languages bolted on afterwards as a top-up. v3.1 adds text and tagged prompts until every writing system carries roughly comparable weight — English excepted, because the prompt format itself is English: the sections, the tags and the descriptions are all written in it, so it stays the majority no matter what.
Measured share of the v3.1 corpus:
| Script | Languages | Tagged prompts | Raw text | Share |
|---|---|---|---|---|
| Latin — base | English: the original corpus, image lots and register lots | base | — | 68.3 % |
| Han | zh | ✓ | ✓ | 6.7 % |
| Hangul | ko | ✓ | ✓ | 6.6 % |
| Latin, accented | fr | ✓ | ✓ | 6.3 % |
| Arabic | ar | ✓ | ✓ (new) | 4.1 % |
| Latin | es, de, it, pt | ✓ | — | 1.3 % each |
| Cyrillic | ru | ✓ | — | 1.3 % |
The rule behind those numbers: a script that inherits nothing from Latin needs raw text; a Latin script only needs tagged prompts, because the alphabet is already covered and roughly 250 tags are enough to attach a language to it. That is why Chinese, Korean and French carry raw lots and Spanish does not.
The one genuinely new lot is raw Arabic — 550 000 characters. Arabic was the last non-Latin script still living on tagged prompts alone.
The training itself was also restarted from scratch rather than topped up. A network keeps the order it
learned in: whatever comes last weighs more, and lowering the learning rate on a top-up run does not
remove that imbalance, it only arbitrates between preserving what was acquired and correcting it. Same
architecture and same hyper-parameters as v3 — hidden 32768, depth 1, tap 24, lr 1e-3, no linear
path — so what the benchmark below compares is the corpus, not the recipe.
What it buys
Phoneme errors against the 32B, averaged over the four files of each generation, the three seeds and compared against the threshold:
| threshold | v3 | v3.1 | ||
|---|---|---|---|---|
| es | 0.0 | 1.9 | 0.5 | −74 % |
| de | 2.7 | 9.8 | 3.6 | −64 % |
| fr | 4.0 | 8.2 | 3.0 | −64 % |
| it | 2.3 | 4.2 | 1.6 | −63 % |
| ru | 10.7 | 21.5 | 13.6 | −37 % |
| ar | 6.3 | 11.1 | 7.6 | −32 % |
| zh | 3.3 | 3.0 | 2.2 | −25 % |
| ja | 6.7 | 7.5 | 6.4 | −14 % |
| ko | 11.3 | 12.8 | 11.5 | −10 % |
| pt | 17.0 | 25.1 | 24.1 | −4 % |
| en | 0.0 | 0.2 | 0.5 | +0.3 |
| mean | 5.8 | 9.6 | 6.8 | −29 % |
The European languages, which v3 only ever saw as a top-up, gain 60 to 74 %. Arabic gains 32 %, which is where the new raw-text lot shows up. Russian gains 37 %.
Raw text does not predict the outcome. Rapported to each language's own threshold, the two groups overlap completely:
| script | raw text | threshold | v3 | v3.1 | × threshold | |
|---|---|---|---|---|---|---|
| zh | Han | ✓ | 3.3 | 3.0 | 2.2 | 0.67 |
| it | Latin | — | 2.3 | 4.2 | 1.6 | 0.68 |
| fr | Latin | ✓ | 4.0 | 8.2 | 3.0 | 0.75 |
| ja | Kana/Kanji | — | 6.7 | 7.5 | 6.4 | 0.96 |
| ko | Hangul | ✓ | 11.3 | 12.8 | 11.5 | 1.01 |
| ar | Arabic | ✓ | 6.3 | 11.1 | 7.6 | 1.20 |
| ru | Cyrillic | — | 10.7 | 21.5 | 13.6 | 1.27 |
| de | Latin | — | 2.7 | 9.8 | 3.6 | 1.34 |
| pt | Latin | — | 17.0 | 25.1 | 24.1 | 1.42 |
Languages with a raw lot run 0.67 to 1.20; languages without run 0.68 to 1.42. Italian, with tagged prompts only, lands second best overall.
Russian is the clearest case. It is the only non-Latin script here with no raw text and nothing to inherit — Japanese borrows kanji from the Chinese lots, Latin scripts borrow the alphabet from English — and it still gains 37 % between v3 and v3.1 on tagged prompts alone, finishing ahead of German and Portuguese, which are Latin. The rule written in the build scripts — 250 tags are enough once the alphabet is covered — evidently extends to Cyrillic, which the Qwen3-VL tokenizer covers natively.
So raw text is what an unseen script needs, not what a language needs. Where a script is already in the tokenizer's reach, tags carry it.
English pays for it, and the bill is half a phoneme out of 77. That is the whole cost of rebalancing: v3 was 0.2 errors, v3.1 is 0.5, both far below anything audible and below what a single seed resolves. Portuguese barely moves, but nothing moves in Portuguese — the reference itself scatters by 17 there.
The net effect is a change of category rather than a better score. v3 sits at 1.46 to 1.77 times the threshold; v3.1 sits at 1.09 to 1.20. From measurably worse than a seed change, to indistinguishable from one.
The problem with every number I published before
A cosine of 0.79, or "23 character errors out of 869" — neither has a scale. Is 23 good? Compared to what? Zero errors is not the right target either, because the 32B does not reproduce itself. Change nothing but the seed and it re-pronounces the sentence differently.
So the reference is not perfection. It is the 32B compared to itself, same prompt, different seed:
| 32B against itself | |
|---|---|
| Speech | 5.8 phonemes out of 75 (7.8 %) |
| Image | 0.9552 SigLIP2 cosine (floor: 0.5313) |
That gap is the unit. Everything below is normalised so that 32B = 100:
- 100 — swapping the encoder moves the output as much as changing the seed
- above 100 — it moves it less
- below 100 — it moves it more
Below that threshold you are no longer measuring the projection. You are measuring the generator.
Files
Put them in ComfyUI/models/clip_projections/.
| File | Encoder | Head | Size | Encoder + projection |
|---|---|---|---|---|
mmh3-4b-ClipProj-v3.1 |
any Qwen3-VL-4B | ridge | 26 MB | 4.6 GB |
mmh3-4b-ClipProj-v3.1-mlp |
any Qwen3-VL-4B | ridge + residual | 481 MB | 5.1 GB |
mmh3-8b-ClipProj-v3.1 |
any Qwen3-VL-8B | ridge | 41 MB | 9.6 GB |
mmh3-8b-ClipProj-v3.1-mlp |
any Qwen3-VL-8B | ridge + residual | 577 MB | 10.1 GB |
The 8B matrices expect 4096 input dimensions instead of 2560; the node checks the width and refuses a mismatch.
The benchmark
Nothing here is a single render. Every figure comes from three seeds — 42, 100 000 and 100 000 000 — chosen far apart so no one can suspect they are correlated.
| volume | |
|---|---|
| Speech | 297 renders — 9 conditionings × 11 languages × 3 seeds |
| Image | 405 renders — 9 conditionings × 15 prompts × 3 seeds |
Speech is scored in phonemes, by ZIPA-CR-large (88 languages, no lexical decoder — it will not silently repair a botched syllable into a real word), against the 32B of the same seed. Distances are Levenshtein throughout.
Image is scored by SigLIP2 so400m, both against the 32B's render and against the prompt itself — on that second axis the 32B is just one column among nine.
Languages: en, fr, es, de, it, pt, ru, ar, zh, ja, ko.
Results
| Conditioning | Speech | ± | Image | Prompt |
|---|---|---|---|---|
| 32B (reference) | 100.0 | — | 100.0 | 100.0 |
8b-ClipProj-v3.1 |
98.8 | ±1.4 | 100.0 | 100.4 |
8b-ClipProj-v3.1-mlp |
98.2 | ±1.2 | 101.4 | 102.3 |
4b-ClipProj-v3.1 |
97.9 | ±1.0 | 100.1 | 99.4 |
4b-ClipProj-v3.1-mlp |
97.8 | ±1.1 | 100.9 | 100.3 |
| v3 ridge / mlp, 4B and 8B | 93.2 – 95.6 | 99.4 – 102.4 | 99.0 – 100.7 |
± is the spread of the score across the three seeds. Two models separated by less than that are not separated at all.
The raw counts behind the speech score
Three metrics, three units, never added together. PER counts phonemes over three seeds on the current protocol; WER and CER count words and characters as Whisper hears them, single-seed on the earlier 0.3 MP protocol. They are listed side by side because they disagree in useful ways — see the language breakdown below.
| PER (3 seeds) | WER (1 seed) | CER (1 seed) | ||
|---|---|---|---|---|
| 32B (reference) | 0 / 2469 | — | 6 / 174 | 6 / 869 |
| 32B against itself | ~193 / 2469 | 7.8 % | — | — |
8b-v3.1 |
211 / 2469 | 8.5 % | 10 / 174 | 14 / 869 |
8b-v3.1-mlp |
222 / 2469 | 9.0 % | 15 / 174 | 28 / 869 |
4b-v3.1-mlp |
230 / 2469 | 9.3 % | 13 / 174 | 23 / 869 |
4b-v3.1 |
232 / 2469 | 9.4 % | 15 / 174 | 30 / 869 |
4b-v3-mlp |
281 / 2469 | 11.4 % | 18 / 174 | 32 / 869 |
8b-v3-mlp |
317 / 2469 | 12.8 % | 17 / 174 | 29 / 869 |
8b-v3 |
326 / 2469 | 13.2 % | 23 / 174 | 46 / 869 |
4b-v3 |
341 / 2469 | 13.8 % | 19 / 174 | 36 / 869 |
The 32B scores 0 on PER by construction — it is the reference. The row below it is the meaningful one: compared to itself on another seed it drifts by about 7.8 %, and the four v3.1 files sit at 8.5 to 9.4 %. The v3 files sit at 11.4 to 13.8 %, clear of that band.
WER and CER rank the files in nearly the same order, which is the point of quoting both: 8b-v3.1 leads
all three metrics, and no v3 file beats any v3.1 file on any of them.
Why some scores exceed 100 — and why that is not "better than the 32B"
The two image columns do not share a reference, and neither exceedance means what it looks like.
Prompt. This axis is cos(image embedding, prompt embedding). The 32B is not the reference here —
it is one column among nine, and its value is set to 100 only to give the scale a fixed point. Nothing
requires it to be the best, and it demonstrably is not: on the prompt asking for a loaf cut in two, it
renders a single piece on two seeds out of three. A projection that follows the description more closely
earns a higher cosine, legitimately.
Image. Here 100 is the 32B against itself, but the comparison is asymmetric: the threshold pits
32B(seed 42) against 32B(seed 7391) — two different draws — while a projection is compared to
32B(seed 42), the same draw. It plays with its reference's seed, so the draw noise is removed on its
side. A score of 101.4 says only closer to that 32B render than two 32B renders are to each other.
And none of it is significant. The nine models span 0.0048 of cosine on the prompt axis, against a within-model standard deviation of 0.024 to 0.029 — five times larger. The paired test over 45 cases calls all eight projections indistinguishable from the 32B, including the one at 102.3. The +2.3 % is real as a measurement and void as a result.
What actually separates
The corpus, not the size and not the head. All four v3.1 land within one point of each other — 97.8 to 98.8 — while ranging from 4.6 to 10.1 GB. All four v3 sit a clear notch below, 93.2 to 95.6, at identical sizes. A 4B v3.1 beats an 8B v3 by four points while weighing half as much.
Nothing separates in image. All nine conditionings, v3 included, are at or above the threshold: 99.4 to 102.4. Swapping the 32B for a 4B changes the picture less than changing the seed does. On this axis the 32B is not a ceiling — 8b-v3.1-mlp scores 102.3 for prompt fidelity, and on one prompt asking for a loaf cut in two, the 32B rendered a single piece on two seeds out of three while the 8B ridge rendered two on all three.
4B against 8B does not separate on general pronunciation. 97.8 against 98.8, for a seed-to-seed spread of ±1.0 to ±1.4. If you need one number: they are the same.
Ridge against MLP does not separate either. The residual buys nothing measurable in speech. It shows up in image prompt fidelity — 102.3 against 100.4 on the 8B — but that axis has its own noise and I would not choose a file on it.
The image measurements in full
Two independent SigLIP2 so400m readings over the same 405 renders. First, resemblance to the 32B's own render, same prompt and same seed — 45 cases per model:
| cosine | std. dev. | % of threshold | worst prompt | |
|---|---|---|---|---|
| 32B against itself | 0.9552 | — | 100.0 | — |
8b-v3-mlp |
0.9653 | 0.0328 | 102.4 | 0.8804 |
8b-v3.1-mlp |
0.9612 | 0.0363 | 101.4 | 0.8808 |
8b-v3 |
0.9594 | 0.0350 | 101.0 | 0.8910 |
4b-v3.1-mlp |
0.9590 | 0.0321 | 100.9 | 0.9038 |
4b-v3-mlp |
0.9588 | 0.0408 | 100.8 | 0.8989 |
4b-v3.1 |
0.9557 | 0.0445 | 100.1 | 0.8766 |
8b-v3.1 |
0.9552 | 0.0448 | 100.0 | 0.8893 |
4b-v3 |
0.9528 | 0.0423 | 99.4 | 0.8861 |
Floor: 0.5313 — two 32B renders sharing no content at all still score that, on style and generator artefacts alone.
The standard deviation settles it. It runs 0.032 to 0.045, while the entire spread from best to worst model is 0.0125. The scatter within one model is three to four times the gap between models. Nothing here is a ranking.
Second, fidelity to the written prompt — an axis where the 32B is one column among nine rather than the reference:
| cosine | std. dev. | base 100 | |
|---|---|---|---|
8b-v3.1-mlp |
0.1504 | 0.0252 | 102.3 |
4b-v3-mlp |
0.1481 | 0.0275 | 100.7 |
8b-v3.1 |
0.1477 | 0.0290 | 100.4 |
4b-v3.1-mlp |
0.1475 | 0.0249 | 100.3 |
| 32B | 0.1471 | 0.0245 | 100.0 |
4b-v3.1 |
0.1462 | 0.0272 | 99.4 |
8b-v3 |
0.1460 | 0.0267 | 99.3 |
8b-v3-mlp |
0.1458 | 0.0236 | 99.2 |
4b-v3 |
0.1456 | 0.0268 | 99.0 |
Floor: −0.0293 — one scene's image against another scene's prompt.
Same verdict, and harder: the spread across all nine models is 0.0048 for a standard deviation of 0.024 to 0.029, five times larger. Four models sit above the 32B and four below, in an order that carries no information.
The two image axes do not even agree with each other: 8b-v3-mlp tops the resemblance table and sits second from last on prompt fidelity. Imitating the 32B and following the prompt are not the same objective — the 32B itself misses prompts.
What counts as an error
One error is one phoneme inserted, deleted or substituted relative to what the 32B pronounced — same prompt, same seed. Levenshtein distance, nothing weighted, nothing forgiven.
There is no dictionary in the loop. ZIPA transcribes sound to IPA and has no lexical decoder, so it will not quietly repair a botched syllable into a real word the way a speech-to-text engine would. What it writes down is what came out of the speaker.
Concretely, on the French line "la lumière de Marseille", seed 42 — the phonemes following d ɛ ("de"):
| heard as | errors on the line | ||
|---|---|---|---|
| 32B | m a ʀ s ɛ j |
Marseille | — |
8b-v3.1 |
m a ʀ s ɛ j |
Marseille | 4 / 71 |
4b-v3.1 |
m a ʀ s ɛ ʀ ɛ |
"marcerre" | 5 / 71 |
4b-v3 |
m a z ɛ ʀ |
"mazer" | 8 / 71 |
Note how little the toponym costs: 4b-v3.1 botches the name outright and pays one phoneme more than 8b-v3.1 over the whole sentence. That is exactly why the aggregate scores cannot settle the proper-noun question, and why it gets its own section below rather than a place in the ranking.
Per language, because the average hides everything
Errors are counted against the 32B of the same seed, averaged over the three seeds. The first two columns are the yardstick: how long the reference is, and how much the 32B differs from itself.
| length | threshold | 8b-v3.1 |
8b-v3.1-mlp |
4b-v3.1 |
4b-v3.1-mlp |
|
|---|---|---|---|---|---|---|
| en | 77 | 0.0 | 0.7 | 0.7 | 0.7 | 0.0 |
| es | 62 | 0.0 | 1.3 | 0.0 | 0.7 | 0.0 |
| de | 97 | 2.7 | 3.0 | 5.0 | 3.7 | 2.7 |
| it | 62 | 2.3 | 0.7 | 0.3 | 4.3 | 1.0 |
| zh | 85 | 3.3 | 3.3 | 1.0 | 2.7 | 2.0 |
| fr | 70 | 4.0 | 1.7 | 1.3 | 5.0 | 4.0 |
| ar | 94 | 6.3 | 5.7 | 4.7 | 7.7 | 12.3 |
| ja | 73 | 6.7 | 6.3 | 6.7 | 5.7 | 7.0 |
| ru | 77 | 10.7 | 14.0 | 16.3 | 14.0 | 10.0 |
| ko | 64 | 11.3 | 11.0 | 15.0 | 10.7 | 9.3 |
| pt | 62 | 17.0 | 22.7 | 23.0 | 22.3 | 28.3 |
Read it against the threshold column, never in absolute terms:
- English and Spanish — the 32B repeats itself phoneme for phoneme. There, a single phoneme of drift is real signal, and all four files stay within one.
- French, Italian, Chinese, Arabic, Japanese — the projections are at or below the 32B's own variance. In French both 8B files land at 1.7 and 1.3 against a threshold of 4.0: closer to the 32B than the 32B is to itself.
- Portuguese, Russian and Korean carry thresholds of 17.0, 10.7 and 11.3 — the reference rewrites a large share of its own pronunciation between seeds. Any single-seed comparison there was measuring the dice.
Where the phoneme metric misleads, and the cross-check that catches it
A high threshold does not mean the speech is bad. It means the phoneme transcriber cannot hold that language still. Cross-checking against Whisper, which reads words rather than sounds, on the same renders:
| ZIPA threshold | ZIPA v3.1 | × threshold | Whisper, 32B | Whisper, v3.1 | |
|---|---|---|---|---|---|
| pt | 17.0 | 24.1 | 1.42 | 0 / 88 | 1.8 / 88 |
| ru | 10.7 | 13.6 | 1.27 | 0 / 85 | 2.8 / 85 |
| ko | 11.3 | 11.5 | 1.01 | 2 / 85 | 3.2 / 85 |
| ja | 6.7 | 6.4 | 0.96 | 2 / 39 | 5.2 / 39 |
| zh | 3.3 | 2.2 | 0.67 | 2 / 31 | 3.5 / 31 |
Portuguese is the worst language by phoneme and one of the best by word — zero character errors for the 32B, 2 % for the v3.1 files. Russian likewise: Whisper transcribes the 32B and two of the projections word for word.
The cause is exactly what makes ZIPA useful elsewhere: it has no lexical decoder. European Portuguese elides and reduces its vowels, Russian has vowel reduction under stress shift — the phonetic realisation moves from one draw to the next while the word does not. ZIPA counts every allophonic variation as an error; Whisper, which recognises the word, sees none. In Japanese and Chinese the bias runs the other way: Whisper is harsher, because one missed ideogram weighs heavily on 31 characters.
Neither metric is sufficient alone. Where the ZIPA threshold is high, read the word column. (Whisper figures are single-seed, on the earlier 0.3 MP protocol.)
The 32B is one of the least stable models here
Distance between two renders of the same model, seed changed, nothing else:
| against itself | against the 32B | |
|---|---|---|
4b-v3.1 |
3.6 | 7.0 |
4b-v3.1-mlp |
4.0 | 7.0 |
8b-v3.1 |
4.1 | 6.4 |
8b-v3.1-mlp |
4.7 | 6.7 |
| 32B | 5.8 | — |
The projections repeat themselves better than the model they imitate.
And the gap to the 32B is reproducible, not random. Each projection sits far closer to itself (3.6–4.7) than to the 32B (6.4–7.0). If swapping the encoder merely added randomness, those two columns would match. They do not — each file redoes the same offset on every seed.
It is not an accent either. An accent would mean one phoneme consistently rendered as another. Counting the actual substitutions says otherwise:
| substitutions | covered by recurring patterns | |
|---|---|---|
| 32B against itself | 103 | ɑ→a ×10, ɾ→r ×6, ʒ→ʐ ×5 |
8b-v3.1 |
103 | 4 % — one pattern |
4b-v3.1 |
114 | 0 % |
8b-v3.1-mlp |
115 | 8 % |
4b-v3.1-mlp |
125 | 0 % |
| the four v3 files | 153–196 | 2–13 % |
The v3.1 files produce as many substitutions as the 32B inflicts on itself — 103 to 125 against 103 — and almost none of them form a repeating pattern. The offset is reproducible but scattered across many different sounds rather than concentrated into a signature. Ironically the clearest patterns belong to the 32B itself, between its own seeds, where they are ordinary allophonic variation.
The v3 files produce 1.5 to 2 times as many.
None of which tells you what it sounds like. A native speaker might well hear something none of these counts describe.
An anecdote, and why it is not a result
This measures nothing. One word, in one language out of eleven, with no denominator — it is recorded here because it was noticed, not because it supports a conclusion. It is deliberately absent from every table above.
The French prompt contains a city name. The four 4B files render it as a non-word on almost every seed, the 8B files render it correctly on all three:
| seed 42 | seed 100 k | seed 100 M | |
|---|---|---|---|
| 32B | ✓ | ✓ | ✓ |
8b-v3.1 |
✓ | ✓ | ✓ |
8b-v3.1-mlp |
✓ | ✓ | ✓ |
4b-v3.1-mlp |
✗ | ✗ | ✓ |
4b-v3.1 |
✗ | ✗ | ✗ |
What keeps it from being a finding, beyond the sample size: it costs almost nothing on the sentence.
4b-v3.1 mangles the name outright and ends up one phoneme worse than 8b-v3.1 over the whole line.
Every aggregate metric in this report is blind to it, which cuts both ways — they cannot confirm it either.
The aggregate scores say 4B and 8B are equivalent, and that is the conclusion to keep. The only reason to mention this at all is that quantisation is independently known to cost factual recall: if your prompts lean on names of people or places, test both sizes on your prompts rather than trusting anything here.
Which one to take
| If you | Take |
|---|---|
| generate images or video without speech | 4b-ClipProj-v3.1 — 4.6 GB, indistinguishable from the 32B |
| are tight on VRAM | 4b-ClipProj-v3.1 — the ridge is 26 MB and gives up nothing measurable |
| generate multilingual speech | 4b-ClipProj-v3.1 covers nine of the eleven languages tested; 8b-ClipProj-v3.1 has the best overall speech score, by less than the seed-to-seed spread |
| have the VRAM to spare | 8b-ClipProj-v3.1 — nothing measured says you need it, nothing says it hurts |
Do not take a v3. That is the only difference this benchmark resolves cleanly: v3 versus v3.1 is real, 4B versus 8B is not, ridge versus residual is not.
How the speech benchmark got affordable
The old protocol rendered full 0.3 MP video and threw the picture away. Measured, same prompt and seed:
| time | |
|---|---|
| 0.3 MP video + audio + previews (old) | 80.3 s |
| same, video VAE and previews removed | 48.7 s |
| 64×64, 8 steps, audio only | 12.4 s |
Verified lossless before adopting: +0.2 dB across every octave band and 1 phoneme out of 70 for dropping the video decode; 64×64 costs 3 phonemes out of 70 against the full-resolution render.
128×128 was rejected — it truncates the start of the sentence, exactly the same eight phonemes at 6 steps and at 8. 64×64 does not. Counter-intuitive, reproducible, and the reason the whole benchmark runs at the smaller size.
That is what made three seeds across 297 renders possible at all: 31 minutes on two cards instead of six and a half hours.
Limitations
Three seeds fix the order of magnitude of the noise, not its tail. Any gap under one point of score is not a result.
The cosine is blind to countable attributes. A whole loaf and a halved loaf, same crust, same paper, same light, give the same vector to the fourth decimal. Image equivalence here means global appearance, not attribute-by-attribute conformity.
Speech quality is deliberately poor. Six to eight steps gives a tinny, canned sound — identically for the 32B, with the same 19 dB dip between 1 and 3 kHz. The benchmark measures correctness of pronunciation, not fidelity of reproduction.
The phoneme metric is unreliable in Portuguese, Russian and Korean — not the speech itself. The reference drifts by 17.0, 10.7 and 11.3 phonemes there between seeds, while Whisper transcribes the same renders with zero to three character errors. Read the word column in those languages.
Quantisation costs facts. Known before, still true, and the most likely explanation for the proper-noun gap.
Licence and responsibility
MIT, like the node. These matrices are derived from the activations of both models and their legal status is unclear; they are provided as-is, for research.
- Qwen3-VL — Alibaba, Apache 2.0.
- MiniMax H3 — custom licence, read it before any commercial use.
Not affiliated with, endorsed by, or connected to Alibaba / Qwen, MiniMax, or Comfy Org. You remain responsible for what you generate.
Credits
Vibe-coded with Anthropic Claude Code (Opus 5). Every number here was measured on this hardware, never estimated. Where a prediction lost to a measurement, the measurement won and the text was rewritten — which happened three times in this release, the largest being a single-seed ranking of the v3.1 files that dissolved entirely once the threshold was known.