How to train a language model from scratch when my dataset is bigger than RAM?
I intend to train a language model from scratch. I have been following the tutorial onHugging Face’s blog. However, when I tried to use their google Colab’s code, I had a memory exceeded error. Their Google Colab’s code uses the LineByLineTextDataset. After digging through the source code, it turns out that LineByLineTextDatasets loads the entire dataset eagerly. No wonder why I had a Memory Exceeded error. My dataset is larger than my RAM capacity.The article hints that if my dataset is bigger than my capacity, I could “opt to load and tokenize examples on the fly, rather than as a preprocessing step.” However, I’m not certain how to achieve this. I will be very grateful if others can points me to the right direction, especially if I can still use the transformers.Trainer.
2020-07-08T18:41:04Z
Do you want to use TensorFlow or PyTorch for the training?
2020-07-08T18:42:36Z
Batik:LineByLineTextDatasetHi@BarikHere are few suggestionYou can modify the LineByLineTextDataset to load each line at a time instead of loading the whole file. So basically your__getitem__should return a single text exampleAt thislinethe examples are encoded into vectors. You can disable this and move the encoding tocollate_batchfunction ofDataCollatorForLanguageModeling.In the collate function you can receive aList[str]instead ofList[torch.Tensor], so take the list of text examples, encode them and then do the maskingI think this will slow down the training, but you can try. Hope this helps.Assuming you are using pytorch
2020-07-08T19:10:15Z
I’m interested in pytorch
2020-07-08T19:42:23Z
To be honest, I am not certain. The blog post hints that we do not need Tensorflow and I successfully reproduced the code after uninstalling Tensorflow in Google Colab (hinting that we’re using PyTorch). However, when I try to reproduce the code in Kaggle, the code only works if I do not uninstall Tensorflow, but maybe that’s because the Kaggle version is reproduced after HF 3.0 is rolled and something internally has changed and now requires Tensorflow
2020-07-08T19:45:28Z
That’s a helpful pointer. I do have a question. Do we know how does theTrainer.__getitem__is normally accessed? If I know it is accessed incrementally from index=0 until the last index, then maybe I can cache 1 million sentences at a time. But if they access it randomly, then the operation will be very expensive I/O wise since I have to constantly move the pointers.
2020-07-08T19:50:38Z
@Batikso the easiest way to overcome all the RAM limitation will likely be to use the newnlp library which is specifically developed for that.Jared, a community contributor, is actually adding right now a loader for text dataset which is probably exactly what you’ll need. You can follow the PR here:https://github.com/huggingface/nlp/pull/356I’ll ping him as well (he’s not yet on the forum I think).
2020-07-08T20:06:15Z
Ran into the same issue as you - TF datasets are greedy by default unless you usetf.data.Dataset.from_generator(), but that can cause performance issues if you’re not careful. I recently opened a PR to the huggingface/nlp library which maps a.txtfile into sharded Apache Arrow formats, which can then be read lazily from disk. So after everything gets merged, you could do something like:from nlp import load_dataset

dset = load_dataset("text", "/path/to/file.txt")["train"]
dset.set_format("tensorflow", columns=["text"])

def dataset_gen():
    for ex in dset:
        yield ex
tf_dataset = tf.data.Dataset.from_generator(dataset_gen, output_types={"text": tf.string})I haven’t tested this exact code, but you get the gist. This should be relatively equivalent to LineByLineTextDataset. Followhttps://github.com/huggingface/nlp/issues/315for more info about dataset efficiency.
2020-07-08T20:13:09Z
To add to@jaredtnielsenanswers I think you shouldn’t have any issue of memory if you are using PyTorch +nlp, it will be fully lazy loading from the drive, almost nothing in RAM (just your current batch, plus the model of course)
2020-07-09T00:16:56Z
Good to know it!
2020-07-31T18:14:00Z
I am going to train a RoBERTa like model on almost 200 GB of Spanish corpus. I will use PyTorch +nlp. Wish me luck!
2020-09-04T17:16:15Z
Best of luck,@mrm8488! Would you be able to share more details on how you did so with the huggingface/NLP library?
2020-09-07T08:26:29Z
I am on that. When I finish it I will share the whole process.
2020-09-10T20:19:53Z
Please share ASAP
2020-09-11T03:01:25Z
Please do, thanks!
2020-09-11T21:01:22Z
Hello! I just wonder if there is an example for training a MLM with TensorFlow + TPU. I don’t want to train from scratch, but rather train for some additional steps on custom data from an existing model. Thank you.
2020-09-16T05:06:43Z
Any updates on this?
2020-09-16T12:21:37Z
@mrm8488we are eagerly waiting for your experiments
2020-09-17T06:08:37Z
Thanks, I’m having the save problem.BTW, I have a question.What is the best practice to train a model from scratch with large datasets like entire Wikipedia?Tweak LineByLineTextDataset ?or use other Dataset classes ?or just use more powerful machine?
2020-09-18T15:24:42Z
Hi guys, you can check the core idea under the hood in this discussion:https://github.com/huggingface/datasets/issues/610#issuecomment-691672919