Does HuBERT need text as well as audio for fine-tuning? / How to achieve sub-5% WER?
There’s a fine-tuning guide provided here that was for wav2vec2:facebook/hubert-xlarge-ll60k · Hugging FaceHowever, I’m interested in achieving the actual performance of wav2vec2 (of 3% WER not 18%). Because this wav2vec2 implementation does not use a language model it suffers at 18%.However, with HuBERT, if I understand correctly, it doesn’t need text?HuBERT: Speech representations for recognition & generationBut the current fine tuning notebook is using a dataset with text.Nevertheless, lets say it does need text. If it is fine tuned will it achieve the same performance or similar in the paper above of around 3% or will it also need its own language model like wav2vec2, and remain at around 18%?
2021-10-08T08:18:06Z
which parts did you change from the Wav2vec2 example to get hubert to work?
2021-10-11T22:50:46Z
Hey,Note that we now have an official fine-tuning example that also works for HuBERT:github.comtransformers/examples/pytorch/speech-recognition at master ·...master/examples/pytorch/speech-recognition🤗 Transformers: State-of-the-art Natural Language Processing for Pytorch, TensorFlow, and JAX. - transformers/examples/pytorch/speech-recognition at master · huggingface/transformersAlso see examples below:github.comtransformers/examples/pytorch/speech-recognition at master ·... - Examplesmaster/examples/pytorch/speech-recognition🤗 Transformers: State-of-the-art Natural Language Processing for Pytorch, TensorFlow, and JAX. - transformers/examples/pytorch/speech-recognition at master · huggingface/transformers
2022-02-17T10:48:10Z
Hey@patrickvonplaten, I’m trying to Fine-tune the HuBERT pretrained model on a custom dataset (multilingual). I’m using a Tokenizer that has the required tokens (Already tested for Wav2Vec2). Do i need to change the feature extractor or the Wav2Vec2FeaureExtractor is the one to use even with HuBERT?
2022-03-18T16:15:02Z
Hey@spranjal25,For multi-lingual fine-tuning I strongly recommend using the XLS-R models, the should perform much betterThink this blog post should help:huggingface.coFine-Tune XLSR-Wav2Vec2 for low-resource ASR with 🤗 TransformersWe’re on a journey to advance and democratize artificial intelligence through open source and open science.