T5 training from scratch
Hi all,I would like to train a T5 model (t5-base version) without loading the pretrained weights, if I write the following:from transformers import T5Config, T5Modelconfig = T5Config.from_pretrained(‘t5-base’)model = T5Model(config)It will produce the t5-base sized T5 model without loading the checkpoint weights?Thanks
2020-11-04T23:16:10Z
sarapapi:from transformers import T5Config, T5Modelconfig = T5Config.from_pretrained(‘t5-base’)model = T5Model(config)Yes, that’s correct. You will definitely see the “pretrained weights” download progress bar in the other case
2020-11-05T09:26:35Z
Thanks for your reply, if I will see the download progress bar it means that the pretrained weights have been loaded?The fact that the T5Config was initialized by “from_pretrained” doesn’t imply downloading something?
2020-11-05T09:50:23Z
That’s only config (eg. hyperparameters)
2020-11-05T19:01:02Z
sarapapi:been loaded?The fact that the T5Config was initialized by “from_pretrained” doesn’t imply downloading something?There will be a little progress bar of < 1mb. The weights are > 1GB.
2020-11-05T19:44:05Z
Yeah, I’ve noticed it. Thank you all!