Size of GPT-2 in fine tuning tutorial
Hi! I’m followingthistutorial to fine tune GPT-2 on wikitext. But the tutorial doesn’t specify what size of GPT-2 they are fine tuning on. I was curious how can I find the size of the model and if there’s any way I can fine-tune it on a different size.
2023-03-27T14:23:03Z
Hi there,The--model_name_or_path=gpt2arg passed to the script indicates that it’s the defaultgpt2model from Huggingface. That would bethisone, which says “This is the smallest version of GPT-2, with 124M parameters.”To change the size of the GPT2 model you’re using, you can pass any of these GPT2 models to that argument:gpt2
gpt2-large
gpt2-medium
gpt2-xlIn general, the models available from Huggingface always have a short name/ID such as “gpt2”, “t5-3b”, etc and you can use that name to look up documentation about that particular model version (how big it is, how it was trained, etc) onhuggingface.co.That being said, I noticed that you linked to Huggingface documentation for version 2.0.0, which is several years old and contains a lot of stuff that’s outdated nowadays. I’d recommend using the documentation and GitHub version from the latest release (4.27.3). That’ll give you access to more models, better features, and it’ll be easier for people in the community to assist you.