Out of CUDA mem when training with streaming dataset
I intend to train my model in a subset of WebSight dataset. However,  when i adjustsubset_sizeinto bigger size, it causes the out of CUDA memory error, though I remain the same batch size of 2.dataset = load_dataset("HuggingFaceM4/WebSight", "v0.2", split='train', streaming=True)
subset_size = 1000
train_rate = 0.8
sub_ds = dataset.take(subset_size)
train_ds = sub_ds.take(round(subset_size * train_rate))
eval_ds = sub_ds.skip(round(subset_size * train_rate))
2024-06-19T16:47:23Z
huggingface.coTroubleshootWe’re on a journey to advance and democratize artificial intelligence through open source and open science.Maybe this will help you, I’m also new
2024-06-19T23:23:25Z
Can we see the training code?