Error in loading pipeline. ValueError: Non-consecutive added token '<|startoftext|>' found

#4
by neogong - opened

Hi Lykon,

Thanks for your great work! I'm following the instruction to load the pipeline:

pipe = AutoPipelineForText2Image.from_pretrained('lykon/dreamshaper-xl-turbo', torch_dtype=torch.float16, variant="fp16")

but it throws an error of ValueError: Non-consecutive added token '<|startoftext|>' found. Should have index 49408 but has index 49406 in saved vocabulary". Do you have any idea on how to solve this issue?

Thank you.

Solved by update transformers and tokenizers to the latest version.

neogong changed discussion status to closed

Sign up or log in to comment