How does the text-generation pipeline know the special stop token?
I use the Llama2 model currently which has the stop token</s>.I tried to change the stop token so that the pipeline would continue to generate regardless of the model predicting the</s>token.I tried to change:model.config.eos_token_id = <someRandomNumber>
# or
tokenizer.eos_token = "somethingelse"but the pipeline still seems to know that it should stop at</s>So I’m asking myself:“How does the the Pipeline (Text-Generation) know what the correct stop token is and does it get the stop token from?”
2024-06-05T10:44:41Z
Hey! You have to change it also in model’s generation config. Try out the below codemodel.generation_config.eos_token_id = None
model.config.eos_token_id = None
2024-06-05T18:26:01Z
Thank you your response! I had no idea about the generation config.The thing is actually my pipeline does not stop to generate.The following print statements print all</s>(the correct stop token).print("1. ", tokenizer.decode(tokenizer.eos_token_id))
print("2. ", tokenizer.decode(model.config.eos_token_id))
print("3. ", tokenizer.decode(model.generation_config.eos_token_id))However, when I create and call the pipeline:pipe = pipeline(task="text-generation", model=model, tokenizer=tokenizer, max_length=200)

# Run text generation pipeline with our next model
while True:
    user_input = "write an sql statement"
    prompt = f"<s>[INTS]{user_input}[/INST]"
    result = pipe(prompt)
    print(result[0]['generated_text'])It outputs:<s>[INTS]write an sql statement[/INST] SELECT * FROM table_name WHERE name = "mario"</s>\n\nWhat does the SQL statement mean?\n\nIn this SQL...You can see that that the correct stop token is set in every single config and the pipeline still ignores the</s>and continues.
2024-06-06T04:57:55Z
Ah sorry, I thought you want to generate even if there is stop token. Then you shouldn’t be setting them toNoneas I showed above.If the model is still generating beyond eos_token set in themodel.generation_config.eos_token_id, can you check if the generation config hasforced_eos_tokensormin_length. Also it would be easier if you provide a reproducible code with the model checkpoint you’re using
2024-06-06T06:30:03Z
No, I’m sorry, this was on me!The script is about a fine tuned LLM (with LoRA). It tries to predict the SQL instruction from a user question and the database table.Here is the entire script:Google ColabLet me give you a quick overview:Loading the base model and merging it with my trained LORA Adapter.Showing all model configsShowing all eos token ids (decoded)Trying out the pipeline with the config filesTrying out the pipeline and setting specifically the</s>stop token (also had no effect)
2024-06-06T12:40:43Z
Thanks a lot for sharing the notebook. Unfortunately I still can’t reproduce it on ‘main’. I am using an arbitrary adapter as I don’t have access to the one you had in Colab, so my wild guess is that there’s smth in the adapter config as the version in Colab is the latest one.Let me know if you can provide the whole script with the adapter, and I will try to find the bug!
2024-06-06T18:06:29Z
No, thank you so much for even trying to help me.But please don’t spend too much time on it but if you can not reproduce it with these script I will go crazy.My trained lora adapter:SQLFormerLlama2_Adapter - Google DriveThe entire training script:Google Colab(If you want to try out the training script you just have to let it run for maybe 3-5min and then you can stop the cell. By then the model will have learned the rough desired structure and will already fail to stop at the stop token)
2024-06-10T05:32:07Z
@VitalContributionhey, I ran the notebook with your adapter and I found the error. The generated “” tokens are not EOS, i.e. the model is generating ids: [829, 29879, 29958]`which are together decoded as “”.I can’t say right now what went wrong with training, or how to train properly since I’m prioritizing other tasks. A workaround here will be to use astop_stringsas a kwarg to pipeline, which stops when a custom string is generated. Seedocsfor more details
2024-06-11T11:03:29Z
This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.