Whisper large-v3 finetuning
I’m trying to finetune whisper model using HuggingFace following this blog postFine-Tune Whisper For Multilingual ASR with 🤗 Transformersand by adding Lora with approximatively 50h of annotated audio. While the finetuning works fine for the medium & large-v2 versions, when i try with the large-v3 the model starts to hallucinate after 400steps, and by hallucination I mean increasing WER with repeating words (i added prediction logs to see how the model reacts) (the training & eval loss are still decreasing). (modifié)
2024-06-18T08:48:57Z
Hello! I have absolutely the same problem, but hallucinations and repetitions start from different number of steps, depends on size of my own datasets. Have you found the solution?
2024-10-23T16:19:38Z
I have the same issue. Fine-tuning the lagre-v3 is very terrible
2024-10-23T16:49:07Z
Is this perhaps a well-known problem?huggingface.coopenai/whisper-large-v3-turbo · Why continue the charade? to get people to...You can't fine-tine a whisper model. This: https://huggingface.co/blog/fine-tune-whisper is only real on paper. Test your models.. Don't rely on metrics.