Do we need to explicity save the model if the save steps is not a multiple of the num steps with HF?
Do we need to explicity save the model if the save steps is not a multiple of the num steps with HF trainera?trainer.train()
    trainer.save_model(output_dir=output_dir)  # TODO is this relaly needed?cross:Discordref:Do we need to explicity save the model if the save steps is not a multiple of the num steps with HF?
2023-10-09T18:18:17Z
Do we need to explicitly save a Hugging Face (HF) model trained with HF trainer after the trainer.train() even if we are checkpointing?related question:huggingface transformers - Do we need to explicitly save a Hugging Face (HF) model trained with HF trainer after the trainer.train() even if we are checkpointing? - Stack Overflow
2023-10-09T19:58:03Z
going to use:# - Make sure to save best checkpoint TODO: do we really need this? https://stackoverflow.com/questions/77261009/do-we-need-to-explicitly-save-a-hugging-face-hf-model-trained-with-hf-trainer
    final_ckpt_dir = output_dir / f'ckpt-{max_steps}'
    final_ckpt_dir.mkdir(parents=True, exist_ok=True)
    trainer.save_model(output_dir=final_ckpt_dir)  # TODO is this relaly needed? https://discuss.huggingface.co/t/do-we-need-to-explicity-save-the-model-if-the-save-steps-is-not-a-multiple-of-the-num-steps-with-hf/56745
    print('Done!\a')
2024-02-06T02:15:10Z
if you want to save tokenizer I think you need to do:tokenizer.save_pretrained(training_args.output_dir)e.g.,# note: seems trainer doesn't save tokenizer automatically https://chat.openai.com/c/c40db3a8-b614-40e0-b492-67319a1807e7 
    trainer.save_model(output_dir=output_dir)  # TODO is this really needed? https://discuss.huggingface.co/t/do-we-need-to-explicity-save-the-model-if-the-save-steps-is-not-a-multiple-of-the-num-steps-with-hf/56745
    ## tokenizer.save_pretrained(output_dir=output_dir)  # ref: https://discuss.huggingface.co/t/do-we-need-to-explicity-save-the-model-if-the-save-steps-is-not-a-multiple-of-the-num-steps-with-hf/56745/3final ref:huggingface transformers - Do we need to explicitly save a Hugging Face (HF) model trained with HF trainer after the trainer.train() even if we are checkpointing? - Stack Overflow
2024-02-06T14:15:37Z
This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.