Run_clm.py stops after some % with error
Hey i’m trying to fine tune a german model. Fine tuning worked previously with the gpt-medium and a small input.txt (around 100kb).Now i try to fine tune dbmdz/german with a dataset of ~3mb (Some fiction books i pasted in the .txt file).after some % i get a wall of text and the following error:“C:/w/b/windows/pytorch/aten/src/ATen/native/cuda/Indexing.cu:662: block: [8,0,0], thread: [88,0,0] AssertionsrcIndex < srcSelectDimSizefailed.C:/w/b/windows/pytorch/aten/src/ATen/native/cuda/Indexing.cu:662: block: [8,0,0], thread: [89,0,0] AssertionsrcIndex < srcSelectDimSizefailed.C:/w/b/windows/pytorch/aten/src/ATen/native/cuda/Indexing.cu:662: block: [8,0,0], thread: [90,0,0] AssertionsrcIndex < srcSelectDimSizefailed.C:/w/b/windows/pytorch/aten/src/ATen/native/cuda/Indexing.cu:662: block: [8,0,0], thread: [91,0,0] AssertionsrcIndex < srcSelectDimSizefailed.C:/w/b/windows/pytorch/aten/src/ATen/native/cuda/Indexing.cu:662: block: [8,0,0], thread: [92,0,0] AssertionsrcIndex < srcSelectDimSizefailed.C:/w/b/windows/pytorch/aten/src/ATen/native/cuda/Indexing.cu:662: block: [8,0,0], thread: [93,0,0] AssertionsrcIndex < srcSelectDimSizefailed.C:/w/b/windows/pytorch/aten/src/ATen/native/cuda/Indexing.cu:662: block: [8,0,0], thread: [94,0,0] AssertionsrcIndex < srcSelectDimSizefailed.C:/w/b/windows/pytorch/aten/src/ATen/native/cuda/Indexing.cu:662: block: [8,0,0], thread: [95,0,0] AssertionsrcIndex < srcSelectDimSizefailed.Traceback (most recent call last):File “run_clm.py”, line 407, inmain()File “run_clm.py”, line 376, in maintrain_result = trainer.train(resume_from_checkpoint=checkpoint)File “E:\anaconda\lib\site-packages\transformers\trainer.py”, line 940, in traintr_loss += self.training_step(model, inputs)File “E:\anaconda\lib\site-packages\transformers\trainer.py”, line 1302, in training_steploss = self.compute_loss(model, inputs)File “E:\anaconda\lib\site-packages\transformers\trainer.py”, line 1334, in compute_lossoutputs = model(**inputs)File “E:\anaconda\lib\site-packages\torch\nn\modules\module.py”, line 889, in _call_implresult = self.forward(*input, **kwargs)File “E:\anaconda\lib\site-packages\transformers\models\gpt2\modeling_gpt2.py”, line 899, in forwardtransformer_outputs = self.transformer(File “E:\anaconda\lib\site-packages\torch\nn\modules\module.py”, line 889, in _call_implresult = self.forward(*input, **kwargs)File “E:\anaconda\lib\site-packages\transformers\models\gpt2\modeling_gpt2.py”, line 689, in forwardinputs_embeds = self.wte(input_ids)File “E:\anaconda\lib\site-packages\torch\nn\modules\module.py”, line 889, in _call_implresult = self.forward(*input, **kwargs)File “E:\anaconda\lib\site-packages\torch\nn\modules\sparse.py”, line 145, in forwardreturn F.embedding(File “E:\anaconda\lib\site-packages\torch\nn\functional.py”, line 1913, in embeddingreturn torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)RuntimeError: CUDA error: device-side assert triggered”I already tried altering --block_size and --per_device_train_batch_size 1 but nothing seems to help.
2021-03-12T16:51:20Z
Ok, it seems to be a problem with the text in the train.txt file. If i swap the text everything is fine. What can it be=
2021-03-12T20:21:25Z
Ok, found a solution. It seems the script doesn’t like when i seperate the texts with <|endoftext|>
2022-08-08T21:11:08Z
@Stefan, I am having this same issue. What did you use to separate the texts?