P-tuned Phi-2 produces open ended, repeating, responses
I’ve worked with the guide frompeft docsonly slightly adjusting the preprocessing function to resolve onebugand substituting phi-2 for the original model.  The trained model produces repeating output, for example:[‘Tweet text :@NYTsupporti have complained a dozen times & yet my papers are still thrown FAR from my door. Why is this so hard to resolve? Label :  complaintaintcomplaintaint’]I’m wondering what could be the cause and if there is anything that can be done about it?
2024-05-24T12:42:06Z
I found out where the problem was. Turns out the eos token set in config doesn’t actually influence what’s going on during inference. It has to be set as an argument of thegeneratemethod. [0] All credit for this solution should go to Eugenio-Schiavoni who first wrote about it ongithub.comtldr:Instead of:model.config.eos_token_id = tokenizer.eos_token_iduse:model.generate(input_tokens, eos_token_id=tokenizer.eos_token_id)[0]When using model.generate, it does not stop at eos_token, but instead continues until the maximum length. · Issue #23175 · huggingface/transformers · GitHub
2024-05-25T00:42:36Z
This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.