How to train gpt-2 from scratch? (no fine-tuning)
Hi, I would like to train GPT-2 from scratch. I don’t want to fine-tuning an existing model, but actually train it from scratch with my own tokenizer. How could I do it?Thanks.
2021-01-23T08:24:18Z
Hi@iamnotapenguin, the place I would start is by adapting the following script for causal language modelling to your dataset:transformers/run_clm.py at master · huggingface/transformers · GitHubThis script allows you to specify both the tokenizer and the model architecture, plus you can do multi-gpu training which is advisable if you’re training from scratch.Hope that helps!
2022-10-20T07:08:36Z
Hi@lewtun,the repo link which you have shared is not available now, can you please shared the updated one?
2022-10-20T08:07:36Z
Hey@yubi-sanprit,I believe he referred to thisscript.
2022-10-21T05:45:34Z
Thank you@IdoAmit198
2022-10-30T00:59:20Z
@IdoAmit198I need to load the trained gpt2 model with above script, can you please related repo url?
2022-10-31T12:19:48Z
Hey@yubi-sanpritYou can specify to load a pretrained gpt2 by passing the flag--model_name_or_pathwith the valuegpt2.You can see some examples to run the script in the repo’sREADME.mdYou can also run the script I referred to with the flag--helpalone to see more helpful information and options to use this script.
2022-11-03T12:02:21Z
Thank you@IdoAmit198
2022-11-07T14:49:42Z
@IdoAmit198I have around 230GB of data, how can I pass this huge amount of data while training from scratch? is there any way to pass via lazy loading i.e. only a chunk of data will go from disk to memory and will be deleted once model will be trained on that chunk.Also, will it detect multiple GPUs, or I will need to specify something for this?
2022-11-08T12:56:00Z
yubi-sanprit:e around 230GB of data, how can I pass this huge amount of data while training from scratch? is there any way to pass via lazy loading i.e. only a chunk of data will go from disk to memory and will be deleted once model will be trained on that chunk.Also, will it detect multiple GPUs, or I will need to specify something for this?From my limited knowledge, I’d say thatrun_clm_no_trainer.pyis more suitable for customisations such as lazy loading.
2022-11-08T14:26:21Z
Regards your big data, I think streaming would be a good option (Load the dataset as IterableDataset). You can read about ithere. If you decided it would fit you, then you can still use therun_clm.pyorrun_clm_no_trainer.pyscripts and just make your own changes to it. For example, when you callload_dataset()you should passstreaming=Trueor verify that when you use your data you don’t use random access (since it’s an iterable dataset).If you run the script as is it shouldn’t detect multiple GPUs and use them. What you’re looking for is a distributed training. There are few ways to do that, I’ll list two I’m familiar with:torch.distributed.launch. In case all your GPUs are in 1 node, you should run something like the follows:python -m torch.distributed.launch--nproc_per_node=#NUM_GPUS_YOU_HAVEYOUR_TRAINING_SCRIPT.py (–arg1 --arg2 --arg3 and all other arguments of your training script)In case you’re using multiple nodes, check out the link for a complete explanation.torchrun. Again, the correct command depends whether your GPUs in 1 node, or multiple nodes. In case of multiple nodes read in the link. In case of 1 node like me, try to follows:torchrun--standalone--nnodes=1`–nproc_per_node=$NUM_GPUS_YOU_HAVEYOUR_TRAINING_SCRIPT.py (–arg1 … train script args…)Hope it helps you buddy.
2022-11-09T02:47:02Z
Thank you@IdoAmit198@cerdwin
2022-11-10T11:22:14Z
@IdoAmit198@cerdwinline_by_line argument is not available with pytorch run_clm.py but it is there in tensorflow script.I don’t want to merge sentences after tokenization, is there any way to do this in pytorch
2022-11-18T05:14:15Z
@IdoAmit198with self-trained GPT-2, when I am generating the text, ti gives the following error, while it works with pre-trained gpt-2from transformers import pipeline, set_seedmy_generator_2 = pipeline(task=‘text-generation’, model=‘checkpoint-24000’,tokenizer=gpt_tokenizer,framework=‘pt’)set_seed(42)text = ‘research paper’my_generator_2(text.lower(),max_length=250,num_return_sequences=2)“num_return_sequences has to be 1, but is 2 when doing greedy search.”
2022-11-21T05:45:08Z
To be honest I’m not familiar with that kind of error since I’m not really usingpipeline.I’d recommend posting a new post details what you’re trying to achieve and the error you get.Good luck
2022-11-21T05:49:26Z
Thank you@IdoAmit198.can you please look into thisissue
2022-12-14T05:21:06Z
In answer to my question on big data size and lazy loading:Transformers dataset dict format and its map method to call any function like tokenisation and grouping is designed to run in batches.It will handle any big data with batch run. So, work with any size big data use convert your dataset in Transformers dataset dict format and map method
2022-12-14T05:26:34Z
@IdoAmit198, In reply to this question:with self-trained GPT-2, when I am generating the text, ti gives the following error, while it works with pre-trained gpt-2from transformers import pipeline, set_seedmy_generator_2 = pipeline(task=‘text-generation’, model=‘checkpoint-24000’,tokenizer=gpt_tokenizer,framework=‘pt’)set_seed(42)text = ‘research paper’my_generator_2(text.lower(),max_length=250,num_return_sequences=2)“num_return_sequences has to be 1, but is 2 when doing greedy search.”Transformers inbuilt pipeline for text generation is giving this error. But following method is working well and giving more than one sequences in return:model = AutoModelForCausalLM.from_pretrained(“checkpoint-80000”)def text_generator(prompt):prompt = prompt.lower()input_ids = gpt_tokenizer(prompt, return_tensors=“pt”).input_idsoutputs = model.generate(input_ids, do_sample=True, max_length=150,min_length=100, temperature=1.5, num_return_sequences=10, top_k=50, top_p=25)output_text = gpt_tokenizer.batch_decode(outputs, skip_special_tokens=True)return output_text