Infinity output from gpt2 model?
New to hugging face. Trying to use gpt2 model to generate some text, why the following code yields infinity result?chatgpt says it might due to EOS not correct. I tried a few but does not help.import torchfrom transformers import GPT2LMHeadModel, GPT2Tokenizermodel = GPT2LMHeadModel.from_pretrained(“gpt2”, torchscript=True).eval()tokenizertokenizer = GPT2Tokenizer.from_pretrained(“gpt2”)in_text = “Lionel Messi is something”in_tokens = torch.tensor(tokenizer.encode(in_text))inference#token_eos= torch.tensor([198]) # line break symboltoken_eos = torch.tensor([50256])out_token = Nonei = 0with torch.no_grad():while out_token != token_eos:logits, _ = model(in_tokens)out_token = torch.argmax(logits[-1, :], dim=0, keepdim=True)in_tokens = torch.cat((in_tokens, out_token), 0)text = tokenizer.decode(in_tokens)print(f’step {i} input: {text}', flush=True)i += 1out_text = tokenizer.decode(in_tokens)print(f’ Input: {in_text}‘)print(f’Output: {out_text}’)
2024-06-22T14:17:39Z
Hey HuangLED,It looks like your code might be getting stuck in an infinite loop becauseout_tokenmay never matchtoken_eos. You can try adding a maximum iteration limit to your loop to prevent it from running indefinitely. Here’s a modified version of your loop:max_iterations = 100  # or any suitable number
i = 0
while out_token != token_eos and i < max_iterations:
    logits, _ = model(in_tokens.unsqueeze(0))  # Add batch dimension
    out_token = torch.argmax(logits[0, -1, :], dim=0, keepdim=True)
    in_tokens = torch.cat((in_tokens, out_token), 0)
    text = tokenizer.decode(in_tokens)
    print(f'step {i} input: {text}', flush=True)
    i += 1
2024-06-22T18:28:54Z
Thanks for the discussion, Matt.I am not sure if it is the right thing to do. A hard cut off like this, will just lead to an incomplete sentence at last.That does not seem a nature response to get I assume?