Results of model.generate are different for different batch sizes of the decode-only model
Hello, I am trying the llama model, which is a code only autoregressive generation model.I try to make my model able to accept input from a batch at a time and generate decoding results. However, I found that inputting samples with a batch size greater than 1 at a time can make the generated results unstable.Specifically, I triedinputs = tokenizer("prompt", return_tensors="pt")
input_ids = inputs["input_ids"].to(device)
input_ids,inputsget(tensor([[   1, 9508]], device='cuda:0'),
 {'input_ids': tensor([[   1, 9508]]), 'attention_mask': tensor([[1, 1]])})Andif tokenizer.pad_token is None:
            tokenizer.add_special_tokens({'pad_token': '[PAD]'})
            model.resize_token_embeddings(len(tokenizer))
inputs_b = tokenizer(["prompt","prompt","prompt"], return_tensors="pt", padding=True).to(device)
input_idsb=inputs_b["input_ids"].to(device)
input_idsb,inputs_bget(tensor([[   1, 9508],
         [   1, 9508],
         [   1, 9508]], device='cuda:0'),
 {'input_ids': tensor([[   1, 9508],
         [   1, 9508],
         [   1, 9508]], device='cuda:0'), 'attention_mask': tensor([[1, 1],
         [1, 1],
         [1, 1]], device='cuda:0')})You can see that the tensor for each item in the input_ids is the same, which is the same as my understanding, because the same words are mapped to the same vector.When I used almost the same method and the same parameters to generate, a strange thing happened, and they returned different Tensors.generation_config = GenerationConfig(
    temperature=1,
    top_p=1,
    top_k=50,
    num_beams=1,
    max_new_tokens=128,
)
with torch.no_grad():
    generation_output = model.generate(
        input_ids=input_ids,
        generation_config=generation_config,
        return_dict_in_generate=True,
        output_scores=True,
    )
generation_outputandgeneration_config = GenerationConfig(
    temperature=1,
    top_p=1,
    top_k=50,
    num_beams=1,
    max_new_tokens=128,
)
with torch.no_grad():
    generation_output = model.generate(
        input_ids=input_idsb,
        generation_config=generation_config,
        return_dict_in_generate=True,
        output_scores=True,
    )
generation_outputFor samples with a batchsize of 1, decoding yields" promptly and efficiently.\nThe Company shall not be liable to the Customer for any loss or damage suffered by the Customer as a result of any delay in the delivery of the Goods (even if caused by the Company's negligence) unless the Customer has given written notice to the Company of the delay within 7 days of the date when the Goods were due to be delivered.\nThe Company shall not be liable to the Customer for any loss or damage suffered by the Customer as a result of any delay in the delivery of the Goods (even if caused by the Company's negligence) unless the"While for samples with a batchsize of 3, decoding yields[' promptly and efficiently.\nThe Company is committed to ensuring that there is no modern slavery or human trafficking in its supply chains or in any part of its business. The Company recognises that it has a responsibility to be proactive in ensuring that modern slavery is not taking place within its business or in its supply chains.\nThe Company is committed to ensuring that there is no modern slavery or human trafficking in its supply chains or in any part of its business.\nThe Company is committed to ensuring that there is no modern slavery or human trafficking in',
 ' promptly and efficiently.\nThe Company is committed to ensuring that there is no modern slavery or human trafficking in its supply chains or in any part of its business. The Company recognises that it has a responsibility to be proactive in ensuring that modern slavery is not taking place within its business or in its supply chains.\nThe Company is committed to ensuring that there is no modern slavery or human trafficking in its supply chains or in any part of its business.\nThe Company is committed to ensuring that there is no modern slavery or human trafficking in',
 ' promptly and efficiently.\nThe Company is committed to ensuring that there is no modern slavery or human trafficking in its supply chains or in any part of its business. The Company recognises that it has a responsibility to be proactive in ensuring that modern slavery is not taking place within its business or in its supply chains.\nThe Company is committed to ensuring that there is no modern slavery or human trafficking in its supply chains or in any part of its business.\nThe Company is committed to ensuring that there is no modern slavery or human trafficking in']I want to know why it makes their results different, that is, why each sample is different from batchsize=1 when batchsize=3, when their input tensor wants to wait. And what can I do to fix it so that they have the same results as when they were batchsize 1 (because they are more stable).
2023-07-24T08:40:54Z
I am facing the same issue.@muellerzrcan you help?
2023-10-15T07:55:32Z
Same observation made. Using Llama-2 with LoRA adaptorThis is my code:tokenizer.padding_side = 'left'
inputs = tokenizer(
    test_instructions,  # len == 8
    return_tensors="pt",
    padding=True,
    truncation=True,
)

inputs = {k: v.to(device) for k, v in inputs.items() if k in ['input_ids', 'attention_mask']}
outputs = model.generate(**inputs,
                                          generation_config=generation_config, max_new_tokens=50, temperature=0,
                                          min_length=2
                                         )
responses = tokenizer.batch_decode(outputs, skip_special_tokens=True)Even though I set min_length to 2, my first instance does not have any generations before the  token.
2023-12-21T07:45:04Z
I’ve encountered the same issue where feeding [[sentence A]] or [[sentence A],[sentence A]] resulted in different outputs. Upon examining the logits of the output token where the difference begins, I discovered that this location had two candidate tokens with similar logit scores. Additionally, when I switched my model back to float32 instead of bfloat16, the inconsistency of outputs disappeared. I suspect that the problem may be due to subtle rounding issues. However, I’m also curious as to why this can result in discrepancies when varying the batch size.@THEATLAS@TopRightExit
2023-12-21T08:03:57Z
cc@joaogantewho had a thread on this
2023-12-22T00:40:10Z
Sounds reasonable
2024-04-14T21:11:05Z
I am also seeing this behavior with a fine-tunedmistralai/Mistral-7B-v0.1and HF Generate. The outputs areslightlydifferent when I increase the batch size from 1 to >1. I am using beam search decoding withdo_sample=False. There’s a partial explanation here:model generate with different batch size but get different results · Issue #23017 · huggingface/transformers · GitHub