KeyError: "Invalid key: slice(0, 1000, None). Please first select a split
Hi,I am trying to train a tokenizer and execute the following line of code:new_tokenizer = tokenizer.train_new_from_iterator(batch_iterator(), vocab_size=25000)Though, when I execute it, I get this error:KeyError: "Invalid key: slice(0, 1000, None). Please first select a split. For example: `my_dataset_dictionary['train'][slice(0, 1000, None)]`. Available splits: ['train']"
2021-08-16T08:31:43Z
Hi ! Yourbatch_iteratormust be iterating on a Dataset object, however it looks like you try to iterate over aDatasetDict(it maps split names to Dataset objects).To fix your code, you just have to replacedatasetbydataset["train"]in your definition ofbatch_iterator.Let me know if that works or if if you have other questions
2021-08-16T09:24:42Z
Hi, thanks for that - it works! I am now coming across this error, sadly:[HELP] RuntimeError: CUDA error - when training my model?BeginnersHello everyone, 
I am encountering an error when training my language model from scratch, having trained a tokenizer beforehand. 
I have just trained my tokenizer from scratch on a WordPiece model like BERT, following this notebook:notebooks/tokenizer_training.ipynb at master · huggingface/notebooks · GitHubI then saved my model using this code: 
new_tokenizer.save_pretrained("/content/drive/MyDrive/my-new-tokenizer")

Thus, the folder structure of my-new-tokenizer looks something like this: 
…
2023-09-11T18:13:32Z
But doesn’t that the trainer is only going to use the train dataset, because the iterator is only returning the train split?