Help with Llama 2 Finetuning Setup
Hello!I’m trying to follow the Llama 2 finetunning example provided by databricks herehttps://github.com/databricks/databricks-ml-examples/blob/b1ca47c058461f7fde214914d53b051990064d94/llm-models/llamav2/llamav2-7b/scripts/fine_tune_deepspeed.py#L94.  However, I ran into the following issue “lib/python3.9/site-packages/transformers/generation/configuration_utils.py”, line 354, in validateraise ValueError(ValueError:do_sampleis set toFalse. However, temperature is set to 0.9 – this flag is only used in sample-based generation modes. Setdo_sample=Trueor unset temperature to continue." on the "    model = transformers.AutoModelForCausalLM.from_pretrained(pretrained_model_name_or_path,torch_dtype=torch.float16,trust_remote_code=True,use_auth_token=True,)" line for the meta-llama/Llama-2-7b-chat-hf model.  I don’t set temperature anywhere in the script.Does anyone have any idea what the issue is?
2023-08-09T04:11:37Z
try setting the temperature parameter to 0.1 while initialising the hugging face pipeline
2023-08-09T06:01:26Z
ValueError:do_sampleis set toFalse. However, temperature is set to 0.9 – this flag is only used in sample-based generation modes. Setdo_sample=Trueor unset temperature to continue.my code is:from transformers import GenerationConfig, LlamaForCausalLM, LlamaTokenizerMODEL_NAME = “meta-llama/Llama-2-7b-chat-hf”bnb_config = BitsAndBytesConfig(load_in_4bit=True,bnb_4bit_use_double_quant=True,bnb_4bit_quant_type=“nf4”,bnb_4bit_compute_dtype=torch.bfloat16,)model = LlamaForCausalLM.from_pretrained(MODEL_NAME,device_map=“auto”,trust_remote_code=True,use_auth_token=True,temperature=0.1,do_sample=True,quantization_config=bnb_config,)tokenizer = LlamaTokenizer.from_pretrained(MODEL_NAME)tokenizer.pad_token = tokenizer.eos_tokeni explicitly change the do_sample=True in configuration_utils.py but didn`t worked
2023-08-09T06:44:06Z
I am having the same issue. It was working without problem until last night.I tried to change the config file and update it by adding do_sample=true but did not work.!pip install -qqq bitsandbytes --progress-bar off!pip install -qqq torch --progress-bar off!pip install -q -U git+https://github.com/huggingface/transformers.gitMODEL_NAME = “meta-llama/Llama-2-7b-chat-hf”model = AutoModelForCausalLM.from_pretrained(MODEL_NAME, device_map=“auto”, quantization_config=bnb_config)Error msg:ValueError                                Traceback (most recent call last)in <cell line: 1>()----> 1 model = AutoModelForCausalLM.from_pretrained(MODEL_NAME, device_map=“auto”, quantization_config=bnb_config)5 frames/usr/local/lib/python3.10/dist-packages/transformers/generation/configuration_utils.py in validate(self)352             )353             if self.temperature != 1.0:→ 354                 raise ValueError(355                     greedy_wrong_parameter_msg.format(flag_name=“temperature”, flag_value=self.temperature)356                 )ValueError:do_sampleis set toFalse. However, temperature is set to 0.9 – this flag is only used in sample-based generation modes. Setdo_sample=Trueor unset temperature to continue.
2023-08-09T06:57:14Z
For anyone looking for a solution, it was an issue with the latest release of hugging face transformers released recently.  Please downgrade to the previous version !pip install git+https://github.com/huggingface/transformers@v4.31-release to fix the issue.
2023-08-09T07:08:34Z
Thank you.Downgrading made it work
2023-08-09T07:19:02Z
Thanks It works
2023-08-09T12:14:52Z
I was using autotrain and got the same error but downgrading transformers didn’t solve the issue.!pip install huggingface_hub autotrain-advanced
!pip install git+https://github.com/huggingface/transformers@v4.31-releaseThis is how I installed the package,Any solution for this
2024-01-30T06:58:51Z
besides downgrade any other solution?
2024-01-30T07:26:48Z
As the error raised onFile "/usr/local/lib/python3.10/dist-packages/transformers/generation/configuration_utils.py", line 560, in save_pretrained
    raise ValueError(
ValueError: The generation config instance is invalid -- `.validate()` throws warnings and/or exceptions. Fix these issues to save the configuration.I just comment the code that raised the exception.
2024-01-30T21:24:05Z
I’m hitting the same error, doesn’tdo regression testing?  This is code is so brittle, I just tried with release version4.38.0.dev0and it still is crapping out.
2024-02-16T19:15:01Z
Downgrading to 4.31 did not solve the issue. Can anyone point me to the fix?
2024-02-27T20:49:26Z
This solution works. Thank you so much!!
2024-03-18T17:25:01Z
I am at version 4.37.2 and I still see the issue. Can anyone let me know, please, if the issue also appears in most recent releases?Is there any fix on its way?I tried all suggestions I could find on this issue, none of them worked.Thanks
2024-03-20T19:54:58Z
The latest transformers version I could get actually working & saving was 4.36.2 (it still shows the warning but doesn’t crash). Downgrading more than that caused error with other things (like not knowing what “mistral” is a model for example).
2024-04-02T02:26:57Z
version 4.39.2 crashes due to this error.anybody encounters this ?
2024-05-20T19:15:55Z
I’m on version 4.40.1 and it still gives the same error. Downgrading doesn’t work for me since it crashes other parts of the code.