Hugging Face's logo

KBLab
/
kb-whisper-tiny

Automatic Speech Recognition
Safetensors
Swedish
whisper
Model card Files Community
Lauler commited on
Commit
306f06f
·
verified ·
1 Parent(s): 9796803

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +61 -3
README.md CHANGED
@@ -6,9 +6,22 @@ license: apache-2.0
6
  datasets:
7
  - KBLab/rixvox-v2
8
  ---
9
- ## KB-Whisper Tiny
10
 
11
- The National Library of Sweden's new Whisper models trained on over 50,000 hours of Swedish speech.
 
 
 
 
 
 
 
 
 
 
 
 
 
12
 
13
  ### Usage
14
 
@@ -41,4 +54,49 @@ generate_kwargs = {"task": "transcribe", "language": "sv"}
41
  res = pipe("audio.mp3",
42
  chunk_length_s=30,
43
  generate_kwargs={"task": "transcribe", "language": "sv"})
44
- ```
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
6
  datasets:
7
  - KBLab/rixvox-v2
8
  ---
9
+ ## KB-Whisper Tiny
10
 
11
+ The National Library of Sweden releases a new suite of Whisper models trained on over 50,000 hours of Swedish speech. In evaluations across [FLEURS](https://huggingface.co/datasets/google/fleurs), [CommonVoice](https://huggingface.co/datasets/mozilla-foundation/common_voice_16_1) and [NST](https://www.nb.no/sprakbanken/en/resource-catalogue/oai-nb-no-sbr-54/), our best performing model reduces the Word Error Rate (WER) by an average of 47% compared to OpenAI's `whisper-large-v3`. The performance of smaller Whisper model sizes on Swedish speech has also substantially improved, with `kb-whisper-small` outperforming `openai/whisper-large-v3` (a model six times its size).
12
+
13
+ | Model size | | FLEURS | CommonVoice | NST |
14
+ |------------|---------|--------|-------------|------|
15
+ | [tiny](https://huggingface.co/KBLab/kb-whisper-tiny) | **KBLab** | **13.2** | **12.9** | **11.2** |
16
+ | | OpenAI | 59.2 | 67.8 | 85.2 |
17
+ | [base](https://huggingface.co/KBLab/kb-whisper-base) | **KBLab** | **9.1** | **8.7** | **7.8** |
18
+ | | OpenAI | 39.6 | 52.1 | 53.4 |
19
+ | [small](https://huggingface.co/KBLab/kb-whisper-small) | **KBLab** | **7.3** | **6.4** | **6.6** |
20
+ | | OpenAI | 20.6 | 26.4 | 26.4 |
21
+ | [medium](https://huggingface.co/KBLab/kb-whisper-medium) | **KBLab** | **6.6** | **5.4** | **5.8** |
22
+ | | OpenAI | 12.1 | 15.8 | 17.1 |
23
+ | [large-v3](https://huggingface.co/KBLab/kb-whisper-large) | **KBLab** | **5.4** | **4.1** | **5.2** |
24
+ | | OpenAI | 7.8 | 9.5 | 11.3 |
25
 
26
  ### Usage
27
 
 
54
  res = pipe("audio.mp3",
55
  chunk_length_s=30,
56
  generate_kwargs={"task": "transcribe", "language": "sv"})
57
+ ```
58
+
59
+ ### Training data
60
+
61
+ Our models have been trained on over 50,000 hours of Swedish audio with text transcriptions. The models were trained in 2 stages, each characterized by the application of different quality filters and thresholds for said filters.
62
+
63
+ Stage 1 employed low threshold values (0.15 to 0.30 BLEU), whereas Stage 2 used stricter thresholds (`BLEU >= 0.7`, weighted ROUGE-N `>= 0.7`, CER of first and last 10 characters `<= 0.2`).
64
+
65
+ | Dataset | Continued pretraining (h) -- Stage 1 | Finetuning (h) -- Stage 2 |
66
+ |-------------|--------------------------|--------------|
67
+ | Subtitles | 34,261 | 3,110 |
68
+ | Riksdag | 21,949 | 5,119 |
69
+ | ISOF | 54 | 54 |
70
+ | NST | 250 | 250 |
71
+ | **Total** | **56,514** | **8,533** |
72
+
73
+ The default when loading our models through Hugging Face is **Stage 2**. We have however also uploaded the checkpoints of our continued pretraing and tagged them. You can load these other checkpoints by specifying the `revision`. For example: [`pretrained-checkpoint`](https://huggingface.co/KBLab/kb-whisper-large/tree/pretrained-checkpoint). The Stage 2 default model's tag is named `standard`.
74
+
75
+ ### Evaluation
76
+
77
+ | Model size | | FLEURS | CommonVoice | NST |
78
+ |------------|---------|--------|-------------|------|
79
+ | [tiny](https://huggingface.co/KBLab/kb-whisper-tiny) | **KBLab** | **13.2** | **12.9** | **11.2** |
80
+ | | OpenAI | 59.2 | 67.8 | 85.2 |
81
+ | [base](https://huggingface.co/KBLab/kb-whisper-base) | **KBLab** | **9.1** | **8.7** | **7.8** |
82
+ | | OpenAI | 39.6 | 52.1 | 53.4 |
83
+ | [small](https://huggingface.co/KBLab/kb-whisper-small) | **KBLab** | **7.3** | **6.4** | **6.6** |
84
+ | | OpenAI | 20.6 | 26.4 | 26.4 |
85
+ | [medium](https://huggingface.co/KBLab/kb-whisper-medium) | **KBLab** | **6.6** | **5.4** | **5.8** |
86
+ | | OpenAI | 12.1 | 15.8 | 17.1 |
87
+ | [large-v3](https://huggingface.co/KBLab/kb-whisper-large) | **KBLab** | **5.4** | **4.1** | **5.2** |
88
+ | | OpenAI | 7.8 | 9.5 | 11.3 |
89
+
90
+
91
+ | Model size | | FLEURS | CommonVoice | NST |
92
+ |------------|---------|--------|-------------|------|
93
+ | tiny | KBLab | **76.6** | **73.7** | **74.3** |
94
+ | | OpenAI | 26.9 | 21.1 | 24.0 |
95
+ | base | KBLab | **83.2** | **79.9** | **78.3** |
96
+ | | OpenAI | 41.1 | 32.5 | 36.9 |
97
+ | small | KBLab | **86.6** | **83.5** | **79.6** |
98
+ | | OpenAI | 64.0 | 56.5 | 58.2 |
99
+ | medium | KBLab | **87.6** | **85.0** | **80.2** |
100
+ | | OpenAI | 77.1 | 70.1 | 68.9 |
101
+ | large-v3 | KBLab | **89.8** | **87.2** | **81.1** |
102
+ | | OpenAI | 84.9 | 79.1 | 75.1 |