Load model from checkpoints occurs degraded performance
Hi, I’m trying to load a pre-trained model from the local checkpoint. However, I get significantly different results when I evaluate the performance on the same validation set used in the training phase.The code I’m using is an example notebook: “https://github.com/huggingface/peft/blob/main/examples/image_classification/image_classification_peft_lora.ipynb”After the training, I save the model using the following:trainer.save_model()trainer.log_metrics(“train”, train_results.metrics)trainer.save_metrics(“train”, train_results.metrics)trainer.save_state()Then, I load the model using:new_model = AutoModelForImageClassification.from_pretrained(‘vit-base-patch16-224-in21k-finetuned-lora/checkpoint-18’)new_trainer = Trainer(new_model, args,eval_dataset=val_ds, tokenizer=image_processor,compute_metrics=compute_metrics, data_collator=collate_fn)new_trainer.evaluate(val_ds)However, two trainers gave me completely different results:{‘eval_loss’: 0.4082214832305908,‘eval_accuracy’: 0.88,‘eval_runtime’: 6.8795,‘eval_samples_per_second’: 72.68,‘eval_steps_per_second’: 0.581,‘epoch’: 2.0}and{‘eval_loss’: 4.637523651123047,‘eval_accuracy’: 0.016,‘eval_runtime’: 6.0346,‘eval_samples_per_second’: 82.856,‘eval_steps_per_second’: 0.663}Thus, it seems I’m not loading the model with random weights. Could you please tell me how to load the architecture and the weights?
2023-07-07T11:46:32Z
Ahh, My problem. I was using LoRA thus the new model can not load the corresponding weights.
2023-07-07T12:10:14Z
For anyone wanna know how to load a pre-trained PEFT model, here is an example:new_model = AutoModelForImageClassification.from_pretrained(‘“google/vit-base-patch16-224-in21k”’,label2id=label2id,id2label=id2label,ignore_mismatched_sizes=True,)lora_model = get_peft_model(new_model, config)lora_model = lora_model.from_pretrained(new_model,model_id=‘vit-base-patch16-224-in21k-finetuned-lora/checkpoint-18’)