What model checkpoint do I use if I trained a Word Piece tokenizer?
Hi,I have just trained my own tokenizer from scratch, which is a Word Piece model like BERT, and I have saved it.From there, I am now wanting to train my own language model from scratch using the tokenizer I trained beforehand.However, referring to the code below, what do I change mymodel_checkpointto?model_checkpoint = "gpt2"
tokenizer_checkpoint = "drive/wordpiece-like-tokenizer"I trained a Word Piece model like BERT, so shouldgpt2be changed to something else?Thanks.
2021-08-30T18:08:28Z
You should change it to"bert-base-cased"for instance.
2021-08-30T18:53:29Z
Thanks. Also, how much training and test data do you recommend for training a language model from scratch?