Unable to use custom dataset when training a tokenizer
Hello,I am following this tutorial here:notebooks/tokenizer_training.ipynb at master · huggingface/notebooks · GitHubSo, using this code, I add my custom dataset:from datasets import load_dataset
dataset = load_dataset('csv', data_files=['/content/drive/MyDrive/mydata.csv'])Then, I use this code to take a look at the dataset:datasetAccess an element:dataset['train'][1]Access a slice directory:dataset['train'][:5]After executing the above code successfully, I try to execute this here:new_tokenizer = tokenizer.train_new_from_iterator(batch_iterator(), vocab_size=25000)However, I get this error:KeyError: "Invalid key: slice(0, 1000, None). Please first select a split. For example: `my_dataset_dictionary['train'][slice(0, 1000, None)]`. Available splits: ['train']"How do I fix this?I am trying to train my own tokenizer, and this seems to be an issue.Any help would be appreciated!
2021-08-10T15:50:25Z
When asking for help on the forum, please paste all relevant code. In this case, you did not past the definition ofbatch_iterator.If you are following the notebook, you did not load one dataset, but several (with a split for train/validation/test) which is why you get this error. You should add thesplit="train"argument when you load your dataset, or adapt the code ofbatch_iteratorto index in your dataset dictionary.
2021-08-11T17:00:50Z
Okay, thanks for that. I have trained my own tokenizer from scratch, so how do I use it in the masked language task?