ValueError: expected sequence of length 1024 at dim 1 (got 507)
I’ve been trying to use run_clm.py from thelanguage modelingexample section to fine tune GPT-neo 125m on a small data set I put together. (around 10 megs txt file) I’ve been trying to use sage maker to speed up the process using the example script given under the GPT-neo-125 train section. The problem I ran into was that I get the following message during training: “ValueError: expected sequence of length 1024 at dim 1 (got 507).” From what I can tell, the run_clm.py function group_texts is supposed to drop text that doesn’t conform to length requirements.I started writing this post feeling at wit’s end, but as it turns out I managed to find a solution to the issue. It turns out that on lines 417-418 of run_clm.py there was a check that was allowing smaller inputs through. I’ve fixed it onmy own branchseeing how I came across this board through my google searches, hopefully, someone else out there will find this useful
2024-05-26T19:17:06Z
Solved my issue, thanks! I was running an old version of run_clm.pyIt has been fixed as of May 2023 for all future readers.