UnicodeDecodeError with xprophetnet-large-wiki100-cased-xglue-qg model
Hi I’m new to the transformer model and when I run this codefrom transformers import ProphetNetTokenizer, ProphetNetForConditionalGeneration, ProphetNetConfig

model = ProphetNetForConditionalGeneration.from_pretrained('microsoft/xprophetnet-large-wiki100-cased-xglue-qg')
tokenizer = ProphetNetTokenizer.from_pretrained('microsoft/xprophetnet-large-wiki100-cased-xglue-qg')
```from https://huggingface.co/microsoft/xprophetnet-large-wiki100-cased-xglue-qg. I got an "UnicodeDecodeError: 'utf-8' codec can't decode byte 0xaf in position 51: invalid start byte" error when I run "tokenizer = ProphetNetTokenizer.from_pretrained('microsoft/xprophetnet-large-wiki100-cased-xglue-qg')"  Can any one help? Thank you!
2021-06-29T16:27:47Z
Slight hack to fix the decoder error - I tried editing ProphetNetTokenizer itself, specifically theload_vocabmethod and changed the encoding to ‘latin-1’.image998×420 40 KBThe edit is madehere, I did it locally so I’m not sure whether this could be done for Colab notebooks.This stackoverflow pagegave some suggestions for a fix, of which thelatin-1encoding seemed to work.Caveat:I haven’t tried using the tokenizer any further than just getting the ProphetNetTokenizer.from_pretrained() step working.It gave a warning of"Special tokens have been added in the vocabulary, make sure the associated word embedding are fine-tuned or trained.", so there may be some extra steps needed after.The overall issue is something that can’t be encoded to utf-8 in the vocabulary used. Hope this helps