How to save custom model to get config.json file
Hello experts,I am trying to write BERT model with my own custom model (adding layer end of BERT).It goes well and I would like to save model to avoid future training.  I am using trainer API and so, there are two ways to save model, right?training_arugments and trainer (save the model under a folder)trainging_arguements  + trainertraining_args = TrainingArguments(output_dir=“./sentiment_transformer/”,logging_dir=‘./sentiment_transformer/logs’,logging_strategy=‘epoch’,logging_steps=10,num_train_epochs=1,per_device_train_batch_size=4,per_device_eval_batch_size=4,learning_rate=5e-6,seed=42,save_strategy=‘epoch’,save_steps=100,evaluation_strategy=‘epoch’,eval_steps=100,load_best_model_at_end=True)trainer = Trainer(model=model,args=training_args,train_dataset=dataset_train,eval_dataset=dataset_test,callbacks=[EarlyStoppingCallback(early_stopping_patience=1)])trainer.train()This type of saving cannot save config.json although other files can be saved. If I am saving model with only BERT model without adding any layers, it can save all files including json.config.trainer.save_model("/folder)This also cannot save json.config file.So, the question is how can we get all completed saved information for custom model ?? because without config.json file we cannot load and reuse the model again. Is there any way to save all included files for custom model saving.Your suggestions will be highly appreciated how to save the custom model.
2024-10-20T08:57:09Z
same question
2024-10-20T13:04:54Z
It is most reliable ifsave_pretrained()can be used.stackoverflow.comsaving finetuned model locallyhuggingface-transformersasked byyulGMon06:51AM - 04 May 22 UTC