Neurlang Whipstr STT (ASR)

A deep learning automatic speech recognition (ASR) system for transcribing speech audio into IPA text using transformer-based sequence-to-sequence models.

Language: Universal (IPA), 74+ languages
Model Github: neurlang/whipstr https://github.com/neurlang/whipstr
Model Dataset: Common Voice 21
Model-Native Sample Rates: 8000 Hz, 16000 Hz, 24000 Hz, 32000 Hz, 48000 Hz
Degraded-Performance Sample Rates: 11025 Hz, 22050 Hz, 44100 Hz
License: GPL v2
Total parameters:
- Encoder: 7 253 472
- Transformer: 99 828 768
- Total (trainable): 107 082 240
- Total (incl. positional-encoding buffers, as stored in safetensors): ~114.8M
CER: 24.89% (75.11% success rate)
- Note: Averaged across all supported languages, works better on higher resource languages
WER: 60.25% (39.75% success rate)
- Note: Averaged across all supported languages, works better on higher resource languages

Architecture

encoder_embed_dim=192, d_model=768, nhead=12, num_encoder_layers=6, num_decoder_layers=6, dim_feedforward=3072, dropout=0.15, stride=1, window_size=11
~2.07× whipstr-medium's 51.7M params, ~7.26× whipstr-base's 14.76M
In Whisper terms: ~107M lands just above Whisper-base (74M), well below Whisper-small (244M)
Note: The CNN encoder (conv1–conv4/pool1–pool3/fc1) is hardcoded regardless of variant — only the final fc2 bottleneck layer scales with output_values (7.22M→7.24M→7.25M base→medium→large). All size scaling across variants is in the transformer, not the acoustic front-end. (Documented upstream; see whipstr_encoder.py.)

Inference code

git clone https://github.com/neurlang/whipstr.git
cd whipstr/
uv run --with torch --with transformers --with phase-spectrogram stt_infer_hf.py --audio /home/m/Downloads/LJ001-0001.wav --model neurlang/ipa-whipstr-large-48khz-cv-21

Inference code (microphone)

uv run --with torch --with transformers --with phase-spectrogram --with sounddevice stt_mic_hf.py --model neurlang/ipa-whipstr-large-48khz-cv-21

Training history

Training has run across multiple sessions, with a mid-run switch from batch_size=1 to batch_size=4 (alongside AMP/mixed-precision and other throughput fixes) and several recoveries from unrelated infrastructure issues (a training-restart bug that silently dropped the encoder's weights, and a disk-full interruption) — none of which reflect problems with the model itself, and all of which the checkpoint history below has been verified to recover cleanly from.

Cumulative compute to date

Run	Batch size	Duration	Batches	Samples
run1	1	2h13m	45,000	45,000
run2	1	57h02m	1,158,000	1,158,000
run3	4	4h12m	20,000	80,000
run4	4	13h22m	120,000	480,000
run5	4	38h36m	341,000	1,364,000
run6	4	41h48m	356,000	1,424,000
Total		157h13m (~6.55 days)	2,040,000	4,551,000

Note: because the training script does not yet persist a global step counter across restarts (only epoch), batch numbers within runs 3–6 are restart-local, not a single continuous count — the table above reports each run's own step count and the wall-clock/sample totals, which are unaffected by that.

Key checkpoints (batch_size=1 phase, original EOS/padding scheme)

Step	WER	CER	Notes
152,000	155.62%	90.17%	Early/noisy phase
182,000	133.84%	86.27%	First real improvement
359,000	128.97%	85.15%	Plateau
507,000	110.89%	69.00%	Breakout
632,000	99.30%	59.48%	Transient language-misidentification failure mode observed (resolved by step 1,071,000)
1,048,000	81.46%	38.83%	Large jump
1,071,000	79.84%	38.80%	Misidentification pattern confirmed resolved
1,158,000	—	—	Last checkpoint before switch to batch_size=4 + EOS/padding-mask fixes

Key checkpoints (batch_size=4 phase, corrected EOS token + padding mask)

Step (restart-local)	WER	CER	Notes
15,000	79.72%	41.16%	Recovered from regime-switch dip
49,000	73.00%	33.24%	First checkpoint to cross ipa-whisper-medium's 33.88% CER benchmark
75,000	73.00%	33.31%	Plateau
179,000	71.84%	31.72%	Breakout begins
203,000	67.67%	29.81%	First sub-30% CER
356,000	60.25%	24.89%	Current best (run6)

Known limitations

Best performance is on higher-resource languages; CER/WER are averaged across all 74+ supported languages and vary substantially by language
A small number of persistently hard utterances (very short references, rare-language samples) contribute disproportionately to the current error rate; typical/common-language utterances already perform noticeably better than the aggregate CER suggests

Downloads last month: 98

Safetensors

Model size

0.1B params

Tensor type

F32