Failed to upload 5GB csv to huggingface dataset
I was trying to upload a dataset of 2 million * 2 string to my dataset but keep getting an “internal error” when it almost complete. any thought why this might happen? Thanks for the advice!
2022-03-05T18:32:54Z
Hi ! Can you try separating your csv file into several files and try again ? Maybe 1GB per file ?
2022-03-05T21:41:45Z
Thanks for the quick reply! I managed to upload a 400MB file to the dataset, now my problem is how to upload 70 of them in bulk? doing it one by one is frustrating…
2022-03-10T09:43:18Z
We’re working on letting users upload an entire folder on the website. In the meantime, you can usegitto upload them all at once (more info on how to usegitto upload a dataset here :Share)