SFTTrainer class - and Training arguements
I an trying to fine Tune Gemma 7B IT for a dataset. The code is being run in kaggle. both an existing run and current fails with this error‘TrainingArguments’ object has no attribute ‘model_init_kwargs’My install block is!pip install -q -U torch --index-urlhttps://download.pytorch.org/whl/cu117!pip install -q -U transformers==“4.38.2”!pip install -q accelerate!pip install -q -iSimple indexbitsandbytes!pip install -q -U datasets!pip install -q -U git+https://github.com/huggingface/trl!pip install -q -U git+https://github.com/huggingface/peftparameter block is below (The exact same code was running two months ago !)peft_config = LoraConfig(lora_alpha=16,lora_dropout=0,r=64,bias=“none”,task_type=“CAUSAL_LM”,target_modules=[“q_proj”, “k_proj”, “v_proj”, “o_proj”,“gate_proj”, “up_proj”, “down_proj”,],)training_arguments = TrainingArguments(output_dir=“logs”,num_train_epochs=5,gradient_checkpointing=True,per_device_train_batch_size=1,gradient_accumulation_steps=8,optim=“paged_adamw_32bit”,save_steps=0,logging_steps=25,learning_rate=2e-4,weight_decay=0.001,fp16=True,bf16=False,max_grad_norm=0.3,max_steps=-1,warmup_ratio=0.03,group_by_length=False,evaluation_strategy=‘steps’,eval_steps = 112,eval_accumulation_steps=1,lr_scheduler_type=“cosine”,report_to=“tensorboard”,)trainer = SFTTrainer(model=model,train_dataset=train_data,eval_dataset=eval_data,peft_config=peft_config,dataset_text_field=“text”,tokenizer=tokenizer,max_seq_length=max_seq_length,args=training_arguments,packing=False,)
2024-05-09T18:07:33Z
used the below to resolveInstall Pytorch & other libraries!pip install “torch==2.1.2” tensorboardInstall Hugging Face libraries!pip install --upgrade“transformers==4.38.2”“datasets==2.16.1”“accelerate==0.26.1”“evaluate==0.4.1”“bitsandbytes==0.42.0”“trl==0.7.11”“peft==0.8.2”
2024-05-10T06:07:43Z
This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.